Fully automated psychological text analysis with Large Language Models: A 16-year single-case email study

This is a replication data for the paper titled: Fully automated psychological text analysis with Large Language Models: A 16-year single-case email study submitted for a blind review. Abstract Recent advances in generative artificial intelligence (AI) have opened new avenues for automated psychological text analysis. This paper presents a fully automated pipeline that leverages a Large Language Model (LLM) with systematic prompt engineering to profile psychological constructs in digital communications. Our methodology applies four established frameworks - Self-Determination Theory, Big Five Personality Traits, Psychological Well-being, and Cognitive Behavioral Therapy - to a longitudinal corpus of personal emails. The pipeline includes internal consistency checks, automated evaluation metrics, and human validation against self-report measures to ensure rigor and interpretability. We implemented the multi-theory framework in a Python-based environment, enabling reproducible analysis of 16 years of emails (N = 25,780) and yielding consistent psychological profiles over time. Results demonstrate that LLM-driven profiling aligns with traditional psychometric assessments, supporting the method’s validity. This work introduces a scalable approach for computational psychometrics that integrates theory-driven prompts with generative AI, providing behavioral scientists with a reproducible tool for psychological assessment from digital traces. The approach exemplifies how advanced AI methods can expand behavioral research methodologies and lays a foundation for future behavioral research tools and reproducible workflows. The zipped file contains five csv files: Email_classification-csv: LLM (GPT-3.5 Turbo) classification of 25,780 emails for four psychological theories: SDT, Big Five, PWB and CBT. SDT_regression_data.csv Big_Five_regression_data.csv PWB_regression_data.csv CBT_regression_data.csv For 2-5 files the dependent variable is monthy percentage share of emails the were assigned a given value for categories of one of the four psychological theories analyzed. Linear regression model has been applied, where dependent variable is the percentage of emails in a specified category that assigned a specific value in this category. For example in Big Five Traits Model, for the Openness category, for each month we calculated percentage of emails that exhibit High or Low openness, or None if the content of the email does not provide enough information to assess whether the specific need is relevant. Two dependent variables were created: Openness-high and Openness-low and regressed on all independent variables. Regressions were not run for the None values. Descriptions of independent variables: - income_index: Person X salary income and consulting fees in a given month, normalized to [0,1]. - card_spending: Person X credit card expenditures in a given month, normalized to [0,1]. - abroad_far: dummy variable set to 1 for months when Person X worked in Central Asia - abroad_near: dummy variable set to 1 when Person X worked in other EU country - death_1_war: variable set to 1 in a month when Person X’ farther in law passed away. In the same month Russia invaded Ukraine. The variable was set to .75 in the following month, and to .5 in the month after that. - death_2: variable set to 1 in a month when Person X’ mother passed away. The variable was set to .75 in the following month, and to .5 in the month after that. - court_case: dummy variable set to 1 for months with the emotionally engaging inheritance court case involving other family members. - BIG4_partner: dummy variable set to 1 for months when Person X worked as a partner in BIG4 accounting firm, which resulted in adopting a professional activity sharply different from the usual Person X habits. - AI_company: dummy variable set to 1 for months when Person X worked as C-level executive at a company specializing in artificial intelligence. - elections: dummy variable set to 1 for months when Person X unsuccessfully run in parliamentary elections - covid_lockdown: dummy variable set to 1 for month where Polish government imposed tough measures during two covid lockdowns. - no_receive: number of different email recipients each month, normalized to [0,1]. - avg_length: average number of words in emails sent each month, normalized to [0,1]. While the email data was collected for January 2008 – March 2014 period, financial data was available from October 2009. There were some months where no emails with more than 10 words were sent, yielding 166 monthly observations used for regressions, before removing outliers. Independent variables were tested for multicollinearity, outlier months were removed, regressions were estimated with robust standard errors, and a range of standard tests were conducted for normality and autocorrelation of residuals, confirming good statistical properties of estimated models. Due to privacy concerns, the email texts cannot be publicly shared. However, the classifications of psychological categories derived from the email texts, along with all other relevant data, are made publicly available in this open access repository, with the consent of email author.