Replication data for: Detecting Model Dependence in Statistical Inference: A Response

Inferences about counterfactuals are essential for prediction, answering \"what if\" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not requ ire sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide. Website反事实推断（counterfactuals）在预测、解答“假设”类问题以及估算因果效应的过程中不可或缺。然而，当所提出的反事实推断与手头现有数据偏差过大时，基于规范设定统计分析得出的结论将不再依托经验证据，而是基于主观推测与便捷却站不住脚的模型假设。遗憾的是，标准统计方法默认模型的正确性，而非揭示模型依赖的程度，因此这类问题往往难以被察觉。我们提出了易于实施的方法，用于评估反事实推断，这类方法无需针对指定的模型类别开展敏感性测试。若某项分析未能通过我们提出的检验，则可得知其核心研究结果对至少部分并非基于经验证据的建模选择具有敏感性。我们运用这些方法，评估了探讨“一国民主程度变化对任意因变量的影响”的海量学术文献，以及针对联合国（UN）建设和平行动影响的独立分析。我们发现，诸多学者在无意间更多基于建模假设而非自身研究数据得出结论。针对部分研究问题而言，历史并未提供足够的信息作为研究指引。相关网站