Provably Efficient Posterior Sampling for Sparse Linear Regression via Measure Decomposition

We consider the problem of sampling from the posterior distribution of a <i>d</i>-dimensional coefficient vector θ, given linear observations y=Xθ+ε. For sparse Bayesian models, such posteriors are in general multimodal, and therefore challenging to sample from. This observation has prompted the exploration of various heuristics that aim at approximating the posterior distribution. In this article, we study a different approach based on decomposing the posterior distribution into a log-concave mixture of simple product measures. This decomposition allows us to reduce sampling from a multimodal distribution of interest to sampling from a log-concave one, which is tractable and has been investigated in detail. We prove that, under mild conditions on the prior, for random designs, such measure decomposition is generally feasible when the number of samples per parameter n/d exceeds a constant threshold. We thus obtain a provably efficient (polynomial time) sampling algorithm in a regime where this was previously not known. Numerical simulations confirm that the algorithm is practical, and reveal that it has attractive statistical properties compared to state-of-the-art methods. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.