Replication Data for: AI-Assisted Conversational Interviewing: Effects on Data Quality and Respondent Experience

Paper: Barari, Soubhik, Jarret Angbazo, Natalie Wang, Leah M. Christian, Elizabeth Dean, Zoe Slowinski, and Brandon Sepulvado. \"Al-Assisted Conversational Interviewing: Effects on Data Quality and User Experience.\" Survey Research Methods (forthcoming). Abstract: Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for Al-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to Al 'chatbots' which use large language models (LLMs) to dynamically probe respondents for elaboration and interactively code open-ended responses to fixed questions developed by human researchers. We assessed the Al chatbot's performance in terms of coding accuracy, response quality, and respondent) experience. Our findings reveal that Al chatbots perform moderately well in live coding] even without survey-specific fine-tuning, despite slightly inflated false positive errors due to respondent acquiescence bias. Open-ended responses were more detailed and informative, but this came at a slight cost to respondent experience. Our findings highlight the feasibility of using Al methods such as chatbots enhanced by LLMs to enhance open-ended data collection in web surveys.