Variational Bayes for high-dimensional linear regression with sparse priors

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference (CAVI) algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package sparsevb.本研究针对稀疏高维线性回归中的贝叶斯模型选择先验，开展了平均场尖峰-板条变分贝叶斯（Variational Bayes, VB）近似方法的研究。在设计矩阵满足相容性条件的前提下，我们推导得到该平均场VB近似的先知不等式，证明其能够以最优速率收敛至稀疏真实模型，并可对响应向量实现最优预测。我们对所提算法的实证表现进行了评估，结果显示其性能可与其他当前最优的贝叶斯变量选择方法相媲美。此外，我们通过数值实验证实，广泛使用的坐标上升变分推断（Coordinate-Ascent Variational Inference, CAVI）算法对参数更新顺序极为敏感，可能导致较差的模型表现。为缓解这一问题，我们提出了一种新颖的优先更新策略，该策略采用数据驱动的更新顺序，在模拟实验中展现出更优的性能。本研究中的变分算法已在R语言包sparsevb中实现。