Selection and Aggregation of Conformal Prediction Sets

Conformal prediction is a generic methodology for finite-sample valid distribution-free prediction. This technique has garnered a lot of attention in the literature partly because it can be applied with any machine learning algorithm that provides point predictions to yield valid prediction regions. Of course, the efficiency (width/volume) of the resulting prediction region depends on the performance of the machine learning algorithm. In the context of point prediction, several techniques (such as cross-validation) exist to select one of many machine learning algorithms for better performance. In contrast, such selection techniques are seldom discussed in the context of set prediction (or prediction regions). In this article, we consider the problem of obtaining the smallest conformal prediction region given a family of machine learning algorithms. We provide two general-purpose selection algorithms and consider coverage as well as width properties of the final prediction region. The first selection method yields the smallest width prediction region among the family of conformal prediction regions for all sample sizes but only has an approximate coverage guarantee. The second selection method has a finite sample coverage guarantee but only attains close to the smallest width. The approximate optimal width property of the second method is quantified via an oracle inequality. As an illustration, we consider the use of aggregation of nonparametric regression estimators in the split conformal method with the absolute residual conformal score. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

共形预测（Conformal prediction）是一种用于实现有限样本有效无分布预测的通用方法论。该技术在学术文献中受到广泛关注，部分原因在于其可与任意提供点预测的机器学习算法结合，以生成有效预测区域。当然，最终所得预测区域的效率（宽度/体积）取决于所采用机器学习算法的性能。在点预测场景下，已有诸多技术（如交叉验证）可从众多机器学习算法中遴选性能更优者。与之相对，在集合预测（或预测区域）的场景中，此类遴选方法却鲜有讨论。本文聚焦于在给定机器学习算法族的前提下，获取最小共形预测区域的问题。我们提出了两种通用遴选算法，并分析了最终预测区域的覆盖率与宽度特性。第一种遴选方法可在所有样本量下，于该共形预测区域族中生成宽度最小的预测区域，但仅能提供近似覆盖率保证；第二种遴选方法具备有限样本覆盖率保证，但仅能获得接近最小宽度的预测结果。第二种方法的近似最优宽度特性可通过先知不等式（oracle inequality）进行量化。作为示例，我们考虑在基于绝对残差共形得分的拆分共形方法中，使用非参数回归估计量的聚合策略。本文的补充材料可在线获取，其中包含了用于复现本研究的相关材料的标准化说明。