Replication Data for: Where to place sensitive questions? Experiments on survey response-order and measures of discriminatory attitudes

In survey experiments, should all covariates be administered before the experimental treatment? Some scholars argue that post-treatment items should never be used as covariates because the treatment could bias the measurement of those items and disrupt experimental randomization. Other scholars argue certain items—specifically sensitive questions measuring prejudice—should not be administered pre-treatment. They argue if asked pre-treatment, these items may prime respondents in ways that will influence how they engage with the experiment treatment, thereby affecting the overall outcome of the experiment. Using evidence from four studies (two original collections) that vary the placement of sensitive items—pre-treatment, post-treatment, or in a separate wave—we find little evidence that the placement of sensitive items influences the measurement of those items, the experimental outcomes, nor heterogeneously affects the outcome conditional on the treatment. However, we find the placement of sensitive items inconsistently affects the experimental outcome by interacting with both the measurement of the items and the experimental treatment condition. Overall, we find these measures to be robust to where they are administered. It may be best to place items pre-treatment to preserve randomization. If researcher have reason to include sensitive moderators post-treatment, they should transparently discuss this choice and the anticipated trade-offs.

在调查实验中，是否应当在实验处理（experimental treatment）前对所有协变量（covariates）进行施测？部分学者认为，绝对不应将处理后条目作为协变量，因为实验处理可能会干扰这些条目的测量结果，破坏实验的随机化设计。另有学者则主张，特定类型的条目——即测量偏见的敏感问题——不应在实验处理前施测。他们认为，若在处理前询问这类问题，可能会对受访者产生启动效应，使其后续对实验处理的反应方式发生改变，进而影响实验的整体结果。本研究依托四项研究（含两项原创调研数据集）展开，这些研究对敏感条目的施测位置进行了差异化设置：处理前、处理后或单独置于一个调查波次（survey wave）中。我们的分析未发现敏感条目的施测位置会对该类条目的测量结果、实验结局产生显著影响，也未发现其会根据实验处理条件对结局产生异质性影响。然而，我们发现敏感条目的施测位置会通过与条目测量及实验处理条件的交互作用，对实验结局产生不一致的影响。总体而言，我们认为这类测量指标在施测位置上具备较强稳健性。为保留实验的随机化设计，将条目置于处理前施测或许是更佳选择。若研究者因研究需求需在处理后纳入敏感调节变量（moderators），则应透明化地阐述该选择及其预期存在的权衡取舍。