Boldness-Recalibration for Binary Event Predictions

Probability predictions are essential to inform decision making across many fields. Ideally, probability predictions are (i) well calibrated, (ii) accurate, and (iii) bold, that is, spread out enough to be informative for decision making. However, there is a fundamental tension between calibration and boldness, since calibration metrics can be high when predictions are overly cautious, that is, non-bold. The purpose of this work is to develop a Bayesian model selection-based approach to assess calibration, and a strategy for boldness-recalibration that enables practitioners to responsibly embolden predictions subject to their required level of calibration. Specifically, we allow the user to pre-specify their desired posterior probability of calibration, then maximally embolden predictions subject to this constraint. We demonstrate the method with a case study on hockey home team win probabilities and then verify the performance of our procedures via simulation. We find that very slight relaxation of calibration probability (e.g., from 0.99 to 0.95) can often substantially embolden predictions when they are well calibrated and accurate (e.g., widening hockey predictions’ range from 26%–78% to 10%–91%).

概率预测对诸多领域的决策制定至关重要。理想情况下，优质的概率预测需满足三项标准：(i) 校准良好（well calibrated），(ii) 预测准确，(iii) 区分度充足（bold），即概率分布足够分散以支撑决策制定。然而，校准（calibration）与区分度之间存在根本性的权衡矛盾：当预测过于保守（即缺乏区分度）时，校准指标往往会表现优异。本研究旨在提出一种基于贝叶斯模型选择（Bayesian model selection）的概率校准评估方法，以及一套兼顾区分度与校准的重校准策略，使得从业者能够在满足自身预设校准要求的前提下，稳妥地提升预测的区分度。具体而言，我们允许用户预先指定其期望的校准后验概率，随后在该约束条件下最大化提升预测的区分度。我们以冰球主队获胜概率预测为案例开展实证研究，并通过仿真实验验证了所提方法的性能。研究发现，当预测本身校准良好且准确时，仅需小幅放宽校准概率阈值（例如从0.99调整至0.95），往往就能显著提升预测的区分度——例如可将冰球预测的概率区间从26%–78%拓展至10%–91%。