Charting New Territory: Using Large Language Models to Enhance Survey Instruments in the Organizational Setting

Scale validation and pre-testing are essential yet resource-intensive stages in survey-based organizational research. To support those stages, we propose a novel strategy and framework using large language models (LLMs) to simulate human responses, which can assist researchers in refining survey instruments before administering them to human participants. To validate our approach, we compare LLM-generated responses with those from a previously validated instrument, where several construct relationships have been well-documented in a comprehensive meta-analysis on the topic of workplace deviance. Our findings indicate that LLMs can generate synthetic data that align with theoretical expectations and prior empirical findings. We also conduct robustness checks across different LLMs and developed a step-by-step guide for implementing our prompting strategy. Finally, we discuss the limitations associated with using LLM-generated synthetic data for research purposes.