Inference for Low-Rank Models Without Estimating the Rank

This article studies the inference about linear functionals of high-dimensional low-rank matrices. While most existing inference methods would require consistent estimation of the true rank, our procedure is robust to rank misspecification, making it a promising approach in applications where rank estimation can be unreliable. We estimate the low-rank spaces using pre-specified weighting matrices, known as diversified projections. A novel statistical insight is that, unlike the usual statistical wisdom that overfitting mainly introduces additional variances, the over-estimated low-rank space also gives rise to a non-negligible bias due to an implicit ridge-type regularization. We develop a new inference procedure and show that the central limit theorem holds as long as the pre-specified rank is no smaller than the true rank. In one of our applications, we study multiple testing with incomplete data in the presence of confounding factors and show that our method remains valid as long as the number of controlled confounding factors is at least as large as the true number, even when no confounding factors are present. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

本文针对高维低秩矩阵的线性泛函推断问题展开研究。当前主流推断方法均要求对真实秩进行相合估计，而本文提出的推断流程对秩误设定具备鲁棒性，在秩估计可靠性欠佳的实际应用场景中极具应用潜力。我们借助预指定的加权矩阵（即多样化投影（diversified projections））对低秩空间进行估计。一项创新性的统计学洞察显示：不同于传统统计认知中“过拟合仅会引入额外方差”的观点，过估计的低秩空间还会因隐式岭型正则化产生不可忽略的偏差。本文提出了全新的推断流程，并证明只要预指定的秩不小于真实秩，中心极限定理即可成立。在一项应用案例中，我们研究了存在混杂因素时的缺失数据多重检验问题，结果表明，即便不存在混杂因素，只要受控混杂因素的数量不少于真实数目，本文方法依然有效。本文的补充材料可在线获取，其中包含复现本研究成果所需材料的标准化说明。