Confidently Comparing Estimates with the c-value

Modern statistics provides an ever-expanding toolkit for estimating unknown parameters. Consequently, applied statisticians frequently face a difficult decision: retain a parameter estimate from a familiar method or replace it with an estimate from a newer or more complex one. While it is traditional to compare estimates using risk, such comparisons are rarely conclusive in realistic settings. In response, we propose the “c-value” as a measure of confidence that a new estimate achieves smaller loss than an old estimate on a given dataset. We show that it is unlikely that a large c-value coincides with a larger loss for the new estimate. Therefore, just as a small <i>p</i>-value supports rejecting a null hypothesis, a large c-value supports using a new estimate in place of the old. For a wide class of problems and estimates, we show how to compute a c-value by first constructing a data-dependent high-probability lower bound on the difference in loss. The c-value is frequentist in nature, but we show that it can provide validation of shrinkage estimates derived from Bayesian models in real data applications involving hierarchical models and Gaussian processes. Supplementary materials for this article are available online.

现代统计学为未知参数的估计提供了日益丰富的工具集。因此，应用统计学家时常面临一项艰难抉择：是保留基于熟悉方法得到的参数估计结果，还是改用更新或更复杂的方法生成的估计结果。传统上常以风险（risk）作为估计量的比较依据，但在实际场景中这类对比往往难以得出确定性结论。为此，我们提出“c值（c-value）”作为一种置信度量，用于衡量在给定数据集上新估计量的损失小于旧估计量的置信程度。我们证明，当新估计量出现更大损失时，很难得到较大的c值。因此，正如较小的p值（p-value）支持拒绝原假设一样，较大的c值支持采用新估计量替代旧估计量。针对广泛的问题与估计场景，我们展示了如何通过先构建损失差值的、与数据相关的高概率下界来计算c值。c值本质上属于频率学派框架，但我们证明，在涉及分层模型与高斯过程（Gaussian processes）的真实数据应用中，它可用于验证由贝叶斯模型导出的收缩估计（shrinkage estimates）。本文的补充材料可在线获取。