Manifold Optimization-Assisted Gaussian Variational Approximation

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference, especially in high-dimensional and large data settings. To control the computational cost, while being able to capture the correlations among the variables, the low rank plus diagonal structure was introduced in the previous literature for the Gaussian covariance matrix. For a specific Bayesian learning task, the uniqueness of the solution is usually ensured by imposing stringent constraints on the parameterized covariance matrix, which could break down during the optimization process. In this article, we consider two special covariance structures by applying the Stiefel manifold and Grassmann manifold constraints, to address the optimization difficulty in such factorization architectures. To speed up the updating process with minimum hyperparameter-tuning efforts, we design two new schemes of Riemannian stochastic gradient descent methods and compare them with other existing methods of optimizing on manifolds. In addition to fixing the identification issue, results from both simulation and empirical experiments prove the ability of the proposed methods of obtaining competitive accuracy and comparable converge speed in both high-dimensional and large-scale learning tasks. Supplementary materials for this article are available online.

高斯变分近似（Gaussian variational approximation）是贝叶斯推断中用于近似后验分布的主流方法，尤其适用于高维和大数据场景。为在控制计算成本的同时捕捉变量间的相关性，既往研究针对高斯协方差矩阵引入了低秩加对角结构。针对特定的贝叶斯学习任务，通常会通过对参数化协方差矩阵施加严格约束来保证解的唯一性，但这类约束在优化过程中可能失效。本文通过施加施蒂费尔流形（Stiefel manifold）与格拉斯曼流形（Grassmann manifold）约束，构建了两种特殊的协方差结构，以解决此类分解架构中的优化难题。为在最小化超参数调优工作量的前提下加快更新过程，本文设计了两种全新的黎曼随机梯度下降（Riemannian stochastic gradient descent）方案，并将其与其他现有流形优化方法进行对比。除解决辨识问题外，仿真与实证实验结果均证明，所提方法在高维和大规模学习任务中均可获得具有竞争力的精度与相当的收敛速度。本文的补充材料可在线获取。