Replication Data for: How To Train Your Stochastic Parrot: Large Language Models for Political Texts

Large language models pretrained on massive corpora of text from the Internet have transformed the way that computer scientists approach natural language processing over the past five years. But such models have yet to see widespread adoption in the social sciences, partly due to their novelty and upfront costs. In this paper, we demonstrate how few-shot prompts to large language models can be effectively applied to a wide range of text-as-data tasks in political science—including sentiment analysis, document scaling, and topic modeling. In a series of pre-registered analyses, this approach outperforms conventional supervised learning methods without the need for extensive data pre-processing or large sets of labeled training data. And performance is comparable to expert and crowd-coding methods at a fraction of the cost. We propose a set of best practices for adapting these models to social science measurement tasks, and develop an open-source software package for researchers.