Robust Validation: Confident Predictions Even When Distributions Shift*

While the traditional viewpoint in machine learning and statistics assumes training and testing samples come from the same population, practice belies this fiction. One strategy—coming from robust statistics and optimization—is thus to build a model robust to distributional perturbations. In this paper, we take a different approach to describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions. We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an f-divergence ball around the training population. The method, based on conformal inference, achieves (nearly) valid coverage in finite samples, under only the condition that the training data be exchangeable. An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it; we develop estimators and prove their consistency for protection and validity of uncertainty estimates under shifts. By experimenting on several large-scale benchmark datasets, including Recht et al.’s CIFAR-v4 and ImageNet-V2 datasets, we provide complementary empirical results that highlight the importance of robust predictive validity.机器学习与统计学领域的传统主流观点假设训练与测试样本取自同一总体，但实际应用场景却打破了这一理想化假设。为此，一种源自鲁棒统计学与优化理论的策略应运而生，即构建对分布扰动具备鲁棒性的模型。本文采用截然不同的研究思路，阐述鲁棒预测推断（robust predictive inference）的具体流程：模型不再仅输出单点预测结果，而是会为其预测结果提供不确定性估计。本文提出一种方法，可生成预测集合（prediction sets），其能够（近乎精确地）在训练总体周围的f散度（f-divergence）球内的任意测试分布上达成正确的覆盖水平。该方法基于共形推断（conformal inference），仅需满足训练数据可交换这一前提条件，即可在有限样本下实现（近乎）有效的覆盖保证。我们的方法论核心环节之一是预估未来数据偏移的预期幅度，并据此构建鲁棒性；我们推导出相应的估计量，并证明了其在数据偏移场景下能够保障防护效果与不确定性估计的有效性。我们在多个大规模基准数据集上开展实验，包括Recht等人提出的CIFAR-v4与ImageNet-V2数据集，并给出了互补的实证结果，凸显了鲁棒预测有效性的重要意义。