Replication Data for: Direction Augmentation in the Evaluation of Armed Conflict Predictions

In many forecasting settings, there is a specific interest in predicting the sign of an outcome variable correctly in addition to its magnitude. For instance, when forecasting armed conflicts, positive and negative logchanges in monthly fatalities represent escalation and de-escalation, respectively, and have very different implications. In the ViEWS forecasting challenge, a prediction competition on state-based violence, a novel evaluation score called targeted absolute deviation with direction augmentation (TADDA) has therefore been suggested, which accounts for both for the sign and magnitude of log-changes. While it has a straightforward intuitive motivation, the empirical results of the challenge show that a no-change model always predicting a log-change of zero outperforms all submitted forecasting models under the TADDA score. We provide a statistical explanation for this phenomenon. Analyzing the properties of TADDA, we find that in order to achieve good scores, forecasters often have an incentive to predict no or only modest log-changes. In particular, there is often an incentive to report conservative point predictions considerably closer to zero than the forecaster's actual predictive median or mean. In an empirical application, we demonstrate that a no-change model can be improved upon by tailoring predictions to the particularities of the TADDA score. We conclude by outlining some alternative scoring concepts.

在诸多预测场景中，研究者不仅关注结果变量的数值量级，同时亦格外重视准确预判其符号（正负性）。例如在武装冲突预测任务中，月度死亡人数的对数变化量为正则代表冲突升级，为负则代表冲突缓和，二者所承载的现实意义截然不同。在针对国家层面暴力事件的预测竞赛ViEWS预测挑战赛中，学界提出了一种名为带方向增强的目标绝对偏差（targeted absolute deviation with direction augmentation, TADDA）的新型评估指标，该指标同时兼顾对数变化量的符号与数值大小。尽管该指标的直观动机清晰明确，但该挑战赛的实证结果显示，在TADDA指标下，始终预测对数变化量为0的不变模型（no-change model）的表现优于所有提交参赛的预测模型。本文针对这一现象给出了统计学层面的解释。通过分析TADDA的指标特性，我们发现，为了获得较高的评估得分，预测者往往倾向于预测无对数变化，或是仅预测幅度较小的对数变化。具体而言，预测者常常会倾向于提交更为保守的点预测值——该值相较于其实际预测的中位数或均值，显著更接近0。在一项实证应用中，我们证明了通过针对TADDA指标的特性调整预测策略，可以对不变模型实现性能优化。最后，本文概述了若干可替代的评估指标思路，作为全文总结。