Confidence calibration in a multi-year geopolitical forecasting competition (CAC)

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

本研究探讨了在预测情境下，置信度和准确度随时间推移的发展过程。尽管过度自信在许多情境下已被研究，但很少有研究考察其在长时间段或重要政策领域的演变。本研究采用了一个独特的数据集，该数据集来自一场为期三年的地缘政治预测锦标赛，成千上万的预测者对数百个事件的结局进行了预测。我们试图将研究成果应用于构建问题、互动和启发式方法，以提升预测的准确性。事实上，预测者的置信度与其准确度大致相符。随着信息的不断更新，准确度得到提升。置信度的增长速度与准确度相近，且良好的校准得以持续。然而，仍存在一定程度的过度自信（3%），尤其是在最为自信的预测中。培训有助于降低过度自信，团队协作则提升了预测的准确度。共同作用下，团队协作和培训将过度自信降至1%。我们的研究结果为在重要领域的情境中，对置信度校准及其随时间发展持审慎乐观态度提供了依据。