Skinny Gibbs: A Consistent and Scalable Gibbs Sampler for Model Selection

We consider the computational and statistical issues for high-dimensional Bayesian model selection under the Gaussian spike and slab priors. To avoid large matrix computations needed in a standard Gibbs sampler, we propose a novel Gibbs sampler called “Skinny Gibbs” which is much more scalable to high-dimensional problems, both in memory and in computational efficiency. In particular, its computational complexity grows only linearly in <i>p</i>, the number of predictors, while retaining the property of strong model selection consistency even when <i>p</i> is much greater than the sample size <i>n</i>. The present article focuses on logistic regression due to its broad applicability as a representative member of the generalized linear models. We compare our proposed method with several leading variable selection methods through a simulation study to show that Skinny Gibbs has a strong performance as indicated by our theoretical work. Supplementary materials for this article are available online.

本文研究了高斯尖峰-平板先验（Gaussian spike and slab priors）下的高维贝叶斯模型选择计算与统计问题。为规避标准吉布斯采样器（standard Gibbs sampler）所需的大规模矩阵运算，我们提出一种名为“瘦型吉布斯采样器（Skinny Gibbs）”的新型吉布斯采样器，其在内存占用与计算效率两方面对高维问题均具备更优的可扩展性。具体而言，其计算复杂度仅随预测变量个数<i>p</i>呈线性增长，且即便在<i>p</i>远大于样本量<i>n</i>的场景下，仍可保持强模型选择一致性的特性。鉴于逻辑回归（logistic regression）作为广义线性模型（generalized linear models）的代表性成员应用广泛，本文以其为研究重点。我们通过模拟实验将所提方法与多款主流变量选择方法进行对比，结果表明，如本研究的理论工作所示，瘦型吉布斯采样器具备优异的性能表现。本文补充材料可在线获取。