Replication Data for: Generalizing Survey Experiments Using Topic Sampling

Scholars have made considerable strides in evaluating and improving the external validity of experimental research. However, little attention has been paid to a crucial aspect of external validity – the topic of study. Researchers frequently develop a general theory and hypotheses (e.g., about policy attitudes), then conduct a study on a specific topic (e.g., environmental attitudes). Yet, the results may vary depending on the topic chosen. In this paper, we develop the idea of topic sampling – rather than studying a single topic, we randomly sample many topics from a defined population. As an application, we combine topic sampling with a classic survey experiment design on partisan cues. Using a hierarchical model, we efficiently estimate the effect of partisan cues for each policy, showing that the size of the effect varies considerably, and predictably, across policies. We conclude with advice on implementing our approach and using it to improve theory testing.学界在评估与提升实验研究的外部效度方面已取得长足进展。然而，学界却鲜少关注外部效度的一个关键维度——研究主题本身。研究者通常会先构建一般性理论与假设（例如围绕政策态度），随后针对某一特定主题（例如环境态度）开展研究，但研究结果可能会因所选主题的不同而产生显著差异。本文提出主题抽样（topic sampling）的研究思路：不再局限于单一主题，而是从预先定义的总体中随机抽取多个主题开展研究。作为应用案例，我们将主题抽样与经典的党派线索（partisan cues）调查实验设计相结合。借助层级模型（hierarchical model），我们可以高效估算每项政策中的党派线索效应，结果表明，不同政策间的效应规模不仅差异显著，且这种差异具备可预测性。文末我们就该研究方法的落地实施与应用于理论检验优化的相关事宜给出了建议。