Manifold Optimization-Assisted Gaussian Variational Approximation

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference, especially in high-dimensional and large data settings. To control the computational cost, while being able to capture the correlations among the variables, the low rank plus diagonal structure was introduced in the previous literature for the Gaussian covariance matrix. For a specific Bayesian learning task, the uniqueness of the solution is usually ensured by imposing stringent constraints on the parameterized covariance matrix, which could break down during the optimization process. In this article, we consider two special covariance structures by applying the Stiefel manifold and Grassmann manifold constraints, to address the optimization difficulty in such factorization architectures. To speed up the updating process with minimum hyperparameter-tuning efforts, we design two new schemes of Riemannian stochastic gradient descent methods and compare them with other existing methods of optimizing on manifolds. In addition to fixing the identification issue, results from both simulation and empirical experiments prove the ability of the proposed methods of obtaining competitive accuracy and comparable converge speed in both high-dimensional and large-scale learning tasks. Supplementary materials for this article are available online.高斯变分近似（Gaussian variational approximation）是贝叶斯推断中用于近似后验分布的主流方法，尤其适用于高维与大数据场景。为在控制计算成本的同时捕捉变量间的相关性，已有研究在高斯协方差矩阵中引入了低秩加对角结构。针对特定贝叶斯学习任务，通常通过对参数化协方差矩阵施加严格约束来保证解的唯一性，但这类约束在优化过程中可能失效。本文针对此类分解架构中的优化难题，通过施加施蒂费尔流形（Stiefel manifold）与格拉斯曼流形（Grassmann manifold）约束，构建了两类特殊的协方差结构。为以最小的超参数调优开销加速更新过程，本文设计了两类黎曼随机梯度下降（Riemannian stochastic gradient descent）新方案，并将其与现有流形优化方法进行对比。除解决辨识问题外，仿真与实证实验结果均证明，所提方法在高维与大规模学习任务中均可获得具有竞争力的精度与相当的收敛速度。本文补充材料可在线获取。