Replication data for: Detecting Model Dependence in Statistical Inference: A Response

Inferences about counterfactuals are essential for prediction, answering "what if" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not requ
ire sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide.
<br /><br /> <a href="http://gking.harvard.edu/files/abs/counterf-abs.shtml" target="_blank">Website</a>

反事实推理（counterfactuals）的推断对于预测、解答"what if"问题以及估算因果效应至关重要。然而，当所提出的反事实推理与现有数据偏差过大时，基于规范统计分析得出的结论将不再依托经验证据，而是基于主观推测以及虽便捷但站不住脚的模型假设。遗憾的是，主流统计方法默认模型的真实性，而非揭示其模型依赖程度，因此该问题往往难以被察觉。我们提出了易于应用的反事实推理评估方法，无需针对指定模型类别进行敏感性测试。若某项分析未能通过我们提出的测试，则可判定其实质性研究结果至少对某些并非基于经验证据的建模选择存在敏感性。我们运用该方法评估了两类体量庞大的学术文献：一类聚焦一国民主程度变化对任意因变量的影响，另一类则单独分析联合国维和行动的效应。我们的研究结果显示，诸多学者在无意间更多地基于建模假设而非实际数据得出结论。对于部分研究问题而言，历史数据并未提供足够的信息作为研究指引。<br /><br /> <a href="http://gking.harvard.edu/files/abs/counterf-abs.shtml" target="_blank">官网</a>