Replication Materials for "Running with Scissors? Integrating GPT Models into Public Policy Research"

This repository contains the replication materials for the research note "Running with Scissors? Integrating GPT Models into Public Policy Research," forthcoming in the Policy Studies Journal. The paper demonstrates how OpenAI's GPT models can be used to semi-automate the annotation of legislative testimony within the Advocacy Coalition Framework, identifying belief–emotion dyads under human supervision. The deposit includes everything needed to reproduce the annotation workflow described in the paper: Code — Python scripts that send a prompt to the OpenAI API and batch-process a folder of text files (src/). Prompt and codebook — the full prompt, including task instructions, belief and emotion definitions, and annotation examples (prompt/). Input data — 75 parsed segments of 52 public testimonies from five Arkansas state legislature hearings (March–April 2021) on House Bill 1570 (files_to_process/). Results — an Excel file with GPT's annotations of 1,611 statements and the accompanying human review (belief- and emotion-agreement judgments), together with a data dictionary (results/). The README documents the full workflow: environment setup, model and parameter configuration (model gpt-4-0613, temperature 0.2, top-p 0.6), and how to run the batch annotation. The code is released under the MIT License; the data and codebook under CC-BY-4.0. Note: because the workflow uses a non-zero temperature, re-running it reproduces comparable results rather than byte-for-byte identical annotations.