Replication data for: When Can History be Our Guide? The Pitfalls of Counterfactual Inference

Inferences about counterfactuals are essential for prediction, answering \"what if\" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effe cts of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide. See also: International Conflict, Causal Inference

反事实推断（counterfactuals）对于预测、回答‘倘若……将会如何’类问题以及估算因果效应均不可或缺。然而，当所设定的反事实情境与现有数据偏差过大时，采用规范设定统计分析得到的结论，将不再以经验证据为依托，而是基于主观臆测与便捷却难以成立的模型假设。遗憾的是，标准统计方法往往默认模型的有效性，而非揭示其模型依赖程度，因此该类问题往往难以被察觉。为此，我们提出了易于实施的评估方法，用于检验反事实情境——此类方法无需针对指定的模型类别开展敏感性检验。若某项分析未能通过我们提出的检验，则可确认其核心结论至少对部分非经验性建模选择具有敏感性。我们运用上述方法，对两大领域的海量学术文献展开评估：其一为一国民主程度变化对任意因变量的影响相关研究，其二为联合国维和行动影响的独立分析。研究结果显示，诸多学者在不经意间，其结论更多依托建模假设而非实际观测数据；针对部分研究问题，历史数据所含信息不足以提供可靠的指引。另见：国际冲突（International Conflict）、因果推断（Causal Inference）