An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

In this work, we propose a novel approximated collapsed variational Bayes approach to model selection in linear regression. The approximated collapsed variational Bayes algorithm offers improvements over mean field variational Bayes by marginalizing over a subset of parameters and using mean field variational Bayes over the remaining parameters in an analogous fashion to collapsed Gibbs sampling. We have shown that the proposed algorithm, under typical regularity assumptions, (a) includes variables in the true underlying model at an exponential rate in the sample size, or (b) excludes the variables at least at the first order rate in the sample size if the variables are not in the true model. Simulation studies show that the performance of the proposed method is close to that of a particular Markov chain Monte Carlo sampler and a path search based variational Bayes algorithm, but requires an order of magnitude less time. The proposed method is also highly competitive with penalized methods, expectation propagation, stepwise AIC/BIC, BMS, and EMVS under various settings. Supplementary materials for the article are available online.

本研究提出一种新颖的近似折叠变分贝叶斯（approximated collapsed variational Bayes）方法，用于线性回归中的模型选择任务。该近似折叠变分贝叶斯算法相较平均场变分贝叶斯（mean field variational Bayes）具有明显改进：其通过对部分参数进行边际化处理，并采用与折叠吉布斯采样（collapsed Gibbs sampling）相似的范式，对剩余参数应用平均场变分贝叶斯方法。我们已证实，在典型正则性假设下，所提算法可实现两类性能表现：(a) 若变量属于真实隐含模型，则其变量选择速率随样本量呈指数级增长；(b) 若变量不属于真实模型，则至少可按一阶速率排除无关变量。仿真实验结果表明，所提方法的性能可与特定马尔可夫链蒙特卡罗（Markov Chain Monte Carlo）采样器及基于路径搜索的变分贝叶斯算法相媲美，但所需时间较其少一个数量级。在多种实验设置下，该方法同样具备较强竞争力，可与惩罚类方法、期望传播（expectation propagation）、逐步AIC/BIC、BMS以及EMVS等方法相较量。本文的补充材料可在线获取。