Automating Social Science: LLMs vs. Human Experts in Variable Relationship Identification

This dataset supports the research described in the associated paper. It includes data and code used to evaluate the ability of LLMs to identify and reason about variable relationships extracted from social psychology meta-analyses. The dataset contains: Judgments: LLM and human expert judgments on 247 variable relationships extracted from 56 social psychology meta-analyses published in 2024. This includes information on whether they correctly identified the presence or absence of linear relationships, difference tests, mediation, and moderation effects.Input Data: The variable information presented to both the LLMs and the human experts.LLM Prompts and Code: The specific prompts used to query the LLMs, along with the code used to run the LLMs and process their outputs. This dataset allows researchers to replicate the findings of the original study, to compare the performance of LLMs and human experts, and to explore the use of LLMs in social psychological research. Detailed information on the data structure, variable definitions, and the prompts used can be found in the README file included in the archive.