Robust Q-Learning

<i>Abstract–</i>Q-learning is a regression-based approach that is widely used to formalize the development of an optimal dynamic treatment strategy. Finite dimensional working models are typically used to estimate certain nuisance parameters, and misspecification of these working models can result in residual confounding and/or efficiency loss. We propose a robust Q-learning approach which allows estimating such nuisance parameters using data-adaptive techniques. We study the asymptotic behavior of our estimators and provide simulation studies that highlight the need for and usefulness of the proposed method in practice. We use the data from the “Extending Treatment Effectiveness of Naltrexone” multistage randomized trial to illustrate our proposed methods. Supplementary materials for this article are available online.

摘要– Q学习（Q-learning）是一类基于回归的方法，被广泛用于形式化构建最优动态治疗方案。现有研究通常采用有限维工作模型来估计部分冗余参数，而此类工作模型的误设可能导致残余混杂与/或效率损失。本文提出一种稳健Q学习方法，可通过数据自适应技术完成上述冗余参数的估计。我们对所提估计量的渐近性质展开了理论分析，并通过模拟研究证实了该方法在实际应用中的必要性与实用价值。随后，我们采用‘延长纳曲酮治疗有效性（Extending Treatment Effectiveness of Naltrexone）’多阶段随机试验的数据集对所提方法进行实例演示。本文的补充材料可在线获取。