Replication Data for: When Wording Becomes Variance: Prompt Specification Error in LLM-Based Consumer Surveys

# Replication Materials: When Wording Becomes Variance — # Prompt Specification Error in LLM-Based Consumer Surveys ## Study overview This repository contains all data files and analysis code required to independently replicate the findings reported in the manuscript submitted to the International Journal of Research in Marketing (IJRM). The study introduces Prompt Specification Error (PSE) as a formally defined component of the Total Survey Error framework and quantifies it across 25,000 LLM-generated survey responses spanning five validated marketing constructs, five open-weight LLMs, and ten demographically realistic personas. ## File inventory | File | Type | Description | |---|---|---| | `personas_10.csv` | Data | Synthetic respondent personas (ACS PUMS 2022) | | `paraphrases_500.csv` | Data | 500 prompt paraphrases (20 items × 25) | | `responses_25000.csv` | Data | Complete response log (primary dataset) | | `variance_decomposition.csv` | Results | REML variance components by construct | | `rc3_bootstrap_ci.csv` | Results | Bootstrap 95% CIs for PSE proportions | | `pse_results.csv` | Results | Prompt optimization phase results | | `selection_history.csv` | Results | Greedy selection log | | `IJRM.ipynb` | Code | Main analysis pipeline | | `IJRM-2.ipynb` | Code | Extended ICC and visualization analyses | | `IJRM-RC.ipynb` | Code | Robustness checks (RC1, RC2) | ## Reproducing the results ### From raw data (responses_25000.csv) Run `IJRM-2.ipynb` and `IJRM-RC.ipynb`. No LM Studio required. All results in Tables 3–7 and Figures 1–2 can be reproduced from `responses_25000.csv` alone. ### Full pipeline replication (including data collection) Requires LM Studio 0.4.12 running locally with all five models loaded: - meta-llama/Meta-Llama-3.1-8B-Instruct (Q6_K GGUF) - mistralai/Mistral-7B-Instruct-v0.3 (Q6_K GGUF) - google/gemma-3-4b-it (Q6_K GGUF) - microsoft/Phi-3.5-mini-instruct (Q6_K GGUF) - Qwen/Qwen2.5-7B-Instruct (Q6_K GGUF) Set server endpoint to `http://localhost:1234/v1` and run `IJRM.ipynb`. ## Software requirements - Python 3.10+ - statsmodels >= 0.14 - sentence-transformers >= 2.2 - pandas, numpy, scipy, matplotlib, pingouin - LM Studio 0.4.12 (data collection only) ## Personas and ACS PUMS The ten personas in `personas_10.csv` were derived from the 2022 American Community Survey Public Use Microdata Sample (ACS PUMS), accessed via the U.S. Census Bureau Data API. No personally identifiable information is included; all records are public-use microdata. ## Seed and determinism All models were queried at temperature=0, seed=42 (where supported). Mistral 7B v0.3 was queried at temperature=0 without a seed parameter (see manuscript footnote 2). Results are exactly reproducible on the same hardware and LM Studio version. ## License CC BY 4.0 — reuse with attribution.