Federated Offline Reinforcement Learning

Evidence-based or data-driven dynamic treatment regimes are essential for personalized medicine, which can benefit from offline reinforcement learning (RL). Although massive healthcare data are available across medical institutions, they are prohibited from sharing due to privacy constraints. Besides, heterogeneity exists in different sites. As a result, federated offline RL algorithms are necessary and promising to deal with the problems. In this article, we propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. The proposed model makes the analysis of the site-level features possible. We design the first federated policy optimization algorithm for offline RL with sample complexity. The proposed algorithm is communication-efficient, which requires only a single round of communication interaction by exchanging summary statistics. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed. Extensive simulations demonstrate the effectiveness of the proposed algorithm. The method is applied to a sepsis dataset in multiple sites to illustrate its use in clinical settings. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

基于证据或数据驱动的动态治疗方案对于个性化医疗至关重要，其可从离线强化学习（offline reinforcement learning，RL）中获益。尽管各医疗机构坐拥海量医疗数据，但受隐私约束，这些数据无法实现共享。此外，不同医疗站点间存在数据异质性。因此，联邦离线强化学习算法成为解决上述问题的必要且极具前景的方案。
本文提出一种多站点马尔可夫决策过程（Markov decision process）模型，该模型可兼容站点间的同质性与异质性效应，使得对站点级特征开展分析成为可能。我们设计了首个带有样本复杂度保证的联邦离线强化学习策略优化算法，该算法具备通信高效性，仅需通过交换汇总统计量完成一轮通信交互即可。我们为所提算法提供了理论保证：所学习策略的次优性速率与数据非分布式场景下的速率相当。
大量仿真实验验证了所提算法的有效性。我们将该方法应用于多站点脓毒症数据集，以展示其在临床场景中的应用价值。本文的补充材料可在线获取，其中包含了可复现研究工作的相关材料的标准化说明。