Generative AI, Policymaking, and Disability Bias

Overview This study utilises Structural Topic Modelling (STM) on LLM-generated policy texts using two models - ChatGPT and DeepSeek - for quantitative text analysis and qualitative thematic analysis to investigate how disability is represented in ChatGPT and DeepSeek-generated synthetic texts and their impact on disability policymaking. The study first identified the underlying themes of the UN CRPD (United Nations Convention on the Rights of Persons with Disabilities) with STM. Next, the study conducted prompt-based audits using ChatGPT and DeepSeek, employing prompt templates that referenced the themes of the UN CRPD. These prompts were used to generate synthetic disability policy texts based on pre-trained models to identify potential inherent biases. Next, computational techniques were applied to the Convention text corpora and LLM outputs for quantitative analysis, using cosine similarity to measure similarity across the datasets. Data 1. CRPD_2007_English_utf8_corrected.txt A text version of the UN CRPD (United Nations Convention on the Rights of Persons with Disabilities) documents, converted from its PDF and encoded in UTF-8. The text file is analysed using Structured Topic Modelling (STM) to uncover its latent topics, associated keywords, and representative texts. The findings are used as the reference for keywords and tasks in prompt generation. 2. prompt_list.csv A comma-separated values (CSV) file containing permutations of roles, document types, and keywords used to generate prompts for the study. The file serves as input for systematic prompt construction, with each record representing a unique prompt configuration for generating responses from the ChatGPT and DeeSeek models. 3. deepseek_combine18Jun2025.csv A comma-separated values (CSV) file contains the text responses generated by DeepSeek in response to the prompts defined in prompt_list.csv. 4. chatgpt_combine17Jun2025.csv A comma-separated values (CSV) file contains the text responses generated by ChatGPT in response to the prompts defined in prompt_list.csv. 5. stm_crpd.rds Structured Topic Modelling (STM) model object generated using the R STM package for UNCRPD documents. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study. 6. stm_deepseek.rds Structured Topic Modelling (STM) model object generated using the R STM package for DeepSeek-generated responses. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study. 7. stm_chatgpt.rds Structured Topic Modelling (STM) model object generated using the R STM package for ChatGPT-generated responses. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study.