Replication Data for: How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for post-treatment variables in statistical models, eliminating observations based on post-treatment criteria, or subsetting the data based on post-treatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on post-treatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental post-treatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real-world data. We conclude by providing applied researchers with recommendations for best practice.

原则上，实验为社会科学家精准估计因果效应（causal effects）提供了一种直接的研究方法。然而，学者们常常在无意间通过对可能受实验操控影响的变量施加条件设定，扭曲了处理效应（treatment effect）的估计结果。典型情形包括在统计模型中控制后处理变量（post-treatment variables）、基于后处理标准剔除观测样本，或是根据后处理变量对数据进行子集划分。尽管这类建模选择旨在解决实验开展过程中遇到的常见问题，但它们可能会对因果效应的估计结果造成偏倚。此外，学界目前在很大程度上尚未认识到后处理变量条件设定所引发的问题，我们的研究显示，本学科顶级期刊中频繁出现采用此类研究范式的实验研究论文。我们通过分析论证了实验后处理偏倚（post-treatment bias）的严重性，并借助可视化分析与真实数据再分析，量化了其可能诱发的扭曲程度。最后，我们为应用研究者提供了最佳实践建议。