Replication Data for: Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process

Models for converting expert-coded data to estimates of latent concepts assume different data generating processes. In this article, we simulate ecologically-valid data according to different assumptions, and examine the degree to which common methods for aggregating expert-coded data 1) recover true values and 2) construct appropriate coverage intervals. We find that hierarchical latent variable models (A-M and IRT) and the mean perform similarly when expert error is low; latent variable techniques outperform the mean when expert error is high. Hierarchical A-M and IRT models generally perform similarly, though IRT models are often more likely to include true values within their coverage intervals. The median and non-hierarchical latent variable modeling techniques perform poorly under most assumed data generating processes.