Replication Data for: A Non-parametric Bayesian Model for Detecting Differential Item Functioning: An Application to Political Representation in the US

A common approach when studying the quality of representation involves comparing the latent preferences of voters and legislators, commonly obtained by fitting an item-response theory (IRT) model to a common set of stimuli. Despite being exposed to the same stimuli, voters and legislators may not share a common understanding of how these stimuli map onto their latent preferences, leading to differential item-functioning (DIF) and incomparability of estimates. We explore the presence of DIF and incomparability of latent preferences obtained through IRT models by re-analyzing an influential survey data set, where survey respondents expressed their preferences on roll call votes that U.S. legislators had previously voted on. To do so, we propose defining a Dirichlet Process prior over item-response functions in standard IRT models. In contrast to typical multi-step approaches to detecting DIF, our strategy allows researchers to fit a single model, automatically identifying incomparable sub-groups with different mappings from latent traits onto observed responses. We find that although there is a group of voters whose estimated positions can be safely compared to those of legislators, a sizeable share of surveyed voters understand stimuli in fundamentally different ways. Ignoring these issues can lead to incorrect conclusions about the quality of representation.

研究表征质量的常用范式，是比较选民与议员的潜在偏好——这类偏好通常通过对一组共同刺激物拟合项目反应理论（item-response theory, IRT）模型获得。尽管二者接触的刺激物完全一致，但选民与议员可能对这些刺激物如何映射至自身潜在偏好存在截然不同的认知，由此引发项目功能差异（differential item-functioning, DIF）与估计值不可比的问题。我们通过重新分析一项具有影响力的调查数据集，探究通过IRT模型得到的潜在偏好是否存在DIF与不可比性：该数据集的调查对象曾就美国议员此前已进行的唱名表决（roll call votes）议题表达自身偏好。为此，我们提出在标准IRT模型中为项目反应函数设置狄利克雷过程（Dirichlet Process）先验。与常规的多步骤DIF检测方法不同，我们的策略允许研究者仅拟合单一模型，即可自动识别出潜在特质映射至观测响应方式存在差异的不可比子群体。我们发现，尽管存在一类选民，其估计立场可与议员的立场安全地进行比较，但仍有相当比例的受访选民以截然不同的方式理解刺激物。忽视此类问题，可能会得出关于表征质量的错误结论。