The Crowdsourced Replication Initiative Participant Survey

The Crowdsourced Replication Initiative (CRI) involved 204 researchers who volunteered to engage in a replication of a well-known study on immigration and social policy preferences. In this project, the participants were surveyed four times between August 20th, 2018 and January 20th, 2019. Survey questions with identifying features have been removed to protect participant anonymity and the data are available in the file cri_survey_long_public with labels or *_nolabs, without. The survey included both objective criteria, such as experience with methods and the substantive topic of the replication, and subjective criteria, such as the participants own beliefs about the hypothesis and immigration in general. In addition, they were asked questions about their time commitment, constraints they faced and some other feedback about the process of crowdsourcing.
As of 2024, we provide data on the participants’ reviews of the other teams’ models. These review scores were initially not directly useable due to some problems with the 4th wave of the participant survey. The participants were given model descriptions that did not always match with the models they should have reflected. However, we have now used these paragraphs to match descriptions. We were able to match roughly 95% of all models. The new data file peer_model_dyad allows users to analyze data that are in participant-model dyad format. These data are linkable to both the participant survey here, and the CRI model specification and results data on Github (https://github.com/nbreznau/CRI). Because of matching and uneven numbers of models per team, there are some participants whose rankings apply to dozens of models and others only a few. The variable descriptions for these data are in the peer_model_dyad_codebook file. We also now provide dyadic data that matches each participant with each model specification produced by their team in df_dyad. These data contain all model specifications and the AME (Average Marginal Effect) produced by that model.