On Optimality of Mallows Model Averaging

In the past decades, model averaging (MA) has attracted much attention as it has emerged as an alternative tool to the model selection (MS) statistical approach. <i>Hansen</i> introduced a Mallows model averaging (MMA) method with model weights selected by minimizing a Mallows’ Cp criterion. The main theoretical justification for MMA is an asymptotic optimality (AOP), which states that the risk/loss of the resulting MA estimator is asymptotically equivalent to that of the best but infeasible averaged model. MMA’s AOP is proved in the literature by either constraining weights in a special discrete weight set or limiting the number of candidate models. In this work, it is first shown that under these restrictions, however, the optimal risk of MA becomes an unreachable target, and MMA may converge more slowly than MS. In this background, a foundational issue that has not been addressed is: When a suitably large set of candidate models is considered, and the model weights are not harmfully constrained, can the MMA estimator perform asymptotically as well as the optimal convex combination of the candidate models? We answer this question in both nested and non-nested settings. In the nested setting, we provide finite sample inequalities for the risk of MMA and show that without unnatural restrictions on the candidate models, MMA’s AOP holds in a general continuous weight set under certain mild conditions. In the non-nested setting, a sufficient condition and a negative result are established for the achievability of the optimal MA risk. Implications on minimax adaptivity are given as well. The results from simulations and real data analysis back up our theoretical findings. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

近数十年来，模型平均（Model Averaging, MA）作为模型选择（Model Selection, MS）统计方法的替代工具，受到了广泛关注。汉森（Hansen）提出了一种马洛斯模型平均（Mallows Model Averaging, MMA）方法，其模型权重通过最小化马洛斯Cp准则进行选取。马洛斯模型平均的核心理论依据为渐近最优性（Asymptotic Optimality, AOP），该性质表明所得模型平均估计量的风险/损失渐近等价于最优但不可行的平均模型的风险/损失。现有文献中，马洛斯模型平均的渐近最优性证明通常通过将权重约束于特定离散权重集，或是限制候选模型的数量来完成。本研究首先证明，在上述约束条件下，模型平均的最优风险实为一个无法达成的目标，且马洛斯模型平均的收敛速度可能慢于模型选择方法。在此背景下，一个尚未被解决的基础性问题是：当考虑规模足够大的候选模型集合，且未对模型权重施加不当约束时，马洛斯模型平均估计量能否渐近达到候选模型最优凸组合的性能？我们分别在嵌套与非嵌套场景下解答了这一问题。在嵌套场景中，我们推导了马洛斯模型平均估计量风险的有限样本不等式，并证明在不对候选模型施加非自然约束的前提下，当满足若干温和条件时，马洛斯模型平均的渐近最优性在一般连续权重集下成立。在非嵌套场景中，我们针对最优模型平均风险的可达成性，分别建立了一个充分条件与一个否定性结论。此外，本文还探讨了其在极小极大适应性上的应用价值。模拟实验与真实数据分析结果佐证了我们的理论发现。本文的补充材料可在线获取，其中包含了可用于复现本研究的标准化材料说明。