Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent

With the fast development of big data, learning the optimal decision rule by recursively updating it and making online decisions has been easier than before. We study the online statistical inference of model parameters in a contextual bandit framework of sequential decision-making. We propose a general framework for an online and adaptive data collection environment that can update decision rules via weighted stochastic gradient descent. We allow different weighting schemes of the stochastic gradient and establish the asymptotic normality of the parameter estimator. Our proposed estimator significantly improves the asymptotic efficiency over the previous averaged SGD approach via inverse probability weights. We also conduct an optimality analysis on the weights in a linear regression setting. We provide a Bahadur representation of the proposed estimator and show that the remainder term in the Bahadur representation entails a slower convergence rate compared to classical SGD due to the adaptive data collection. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

随着大数据的快速发展，通过递归更新最优决策规则并开展在线决策来学习最优决策准则，已较以往更为简便。本文聚焦序贯决策场景下的上下文老虎机（contextual bandit）框架，开展模型参数的在线统计推断研究。我们提出了一种适用于在线自适应数据采集环境的通用框架，该框架可通过加权随机梯度下降（stochastic gradient descent，SGD）更新决策规则。我们支持针对随机梯度的多种加权方案，并证明了参数估计量的渐近正态性。相较于此前基于逆概率加权的平均随机梯度下降方法，本文提出的估计量在渐近效率上有显著提升。我们还在线性回归场景下对加权方案开展了最优性分析。我们给出了所提估计量的巴哈杜尔表示（Bahadur representation），并证明由于采用了自适应数据采集方式，巴哈杜尔表示中的余项相较于经典随机梯度下降拥有更慢的收敛速度。本文的补充材料可在线获取，其中包含可用于复现研究成果的标准化材料说明。