Generative AI, Policymaking, and Disability Bias

Overview This study utilises Structural Topic Modelling (STM) on LLM-generated policy texts using two models - ChatGPT and DeepSeek - for quantitative text analysis and qualitative thematic analysis to investigate how disability is represented in ChatGPT and DeepSeek-generated synthetic texts and their impact on disability policymaking. The study first identified the underlying themes of the UN CRPD (United Nations Convention on the Rights of Persons with Disabilities) with STM. Next, the study conducted prompt-based audits using ChatGPT and DeepSeek, employing prompt templates that referenced the themes of the UN CRPD. These prompts were used to generate synthetic disability policy texts based on pre-trained models to identify potential inherent biases. Next, computational techniques were applied to the Convention text corpora and LLM outputs for quantitative analysis, using cosine similarity to measure similarity across the datasets. Data 1. CRPD_2007_English_utf8_corrected.txt A text version of the UN CRPD (United Nations Convention on the Rights of Persons with Disabilities) documents, converted from its PDF and encoded in UTF-8. The text file is analysed using Structured Topic Modelling (STM) to uncover its latent topics, associated keywords, and representative texts. The findings are used as the reference for keywords and tasks in prompt generation. 2. prompt_list.csv A comma-separated values (CSV) file containing permutations of roles, document types, and keywords used to generate prompts for the study. The file serves as input for systematic prompt construction, with each record representing a unique prompt configuration for generating responses from the ChatGPT and DeeSeek models. 3. deepseek_combine18Jun2025.csv A comma-separated values (CSV) file contains the text responses generated by DeepSeek in response to the prompts defined in prompt_list.csv. 4. chatgpt_combine17Jun2025.csv A comma-separated values (CSV) file contains the text responses generated by ChatGPT in response to the prompts defined in prompt_list.csv. 5. stm_crpd.rds Structured Topic Modelling (STM) model object generated using the R STM package for UNCRPD documents. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study. 6. stm_deepseek.rds Structured Topic Modelling (STM) model object generated using the R STM package for DeepSeek-generated responses. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study. 7. stm_chatgpt.rds Structured Topic Modelling (STM) model object generated using the R STM package for ChatGPT-generated responses. The object stores the estimated topic model, including topic-word distributions, document-topic distributions, model parameters, and associated metadata needed to reproduce the analyses in the study.研究概述 本研究针对大语言模型（Large Language Model，LLM）生成的政策文本，采用ChatGPT与DeepSeek两款模型开展结构主题建模（Structural Topic Modelling，STM），结合定量文本分析与定性主题分析，探究ChatGPT与DeepSeek生成的合成文本中残障群体的表征方式及其对残障政策制定的影响。 本研究首先通过STM识别《联合国残疾人权利公约》（United Nations Convention on the Rights of Persons with Disabilities，UN CRPD）的潜在主题。随后，基于参考UN CRPD主题设计的提示词模板，研究使用ChatGPT与DeepSeek开展基于提示词的审计工作，通过预训练模型生成合成残障政策文本，以识别其中潜在的固有偏见。后续，研究将计算技术应用于公约文本语料库与大语言模型输出，开展定量分析，采用余弦相似度衡量各数据集间的文本相似性。 数据集详情 1. CRPD_2007_English_utf8_corrected.txt 该文件为《联合国残疾人权利公约》（UN CRPD）文档的文本版本，由PDF格式转换而来，采用UTF-8编码。研究通过结构主题建模（STM）对该文本文件进行分析，以挖掘其潜在主题、关联关键词与代表性文本，所得结果将作为提示词生成阶段关键词与任务的参考依据。 2. prompt_list.csv 该逗号分隔值（Comma-Separated Values，CSV）文件包含角色、文档类型与关键词的排列组合，用于为本研究生成提示词。该文件作为系统化提示词构建的输入文件，每条记录对应一种用于从ChatGPT与DeepSeek模型获取响应的唯一提示词配置。 3. deepseek_combine18Jun2025.csv 该逗号分隔值（CSV）文件存储了DeepSeek针对prompt_list.csv中定义的提示词所生成的文本响应。 4. chatgpt_combine17Jun2025.csv 该逗号分隔值（CSV）文件存储了ChatGPT针对prompt_list.csv中定义的提示词所生成的文本响应。 5. stm_crpd.rds 该文件为使用R语言STM包针对UN CRPD文档生成的结构主题建模（STM）模型对象，存储了估计得到的主题模型，包括主题-词分布、文档-主题分布、模型参数，以及复现本研究分析所需的相关元数据。 6. stm_deepseek.rds 该文件为使用R语言STM包针对DeepSeek生成的响应文本生成的结构主题建模（STM）模型对象，存储了估计得到的主题模型，包括主题-词分布、文档-主题分布、模型参数，以及复现本研究分析所需的相关元数据。 7. stm_chatgpt.rds 该文件为使用R语言STM包针对ChatGPT生成的响应文本生成的结构主题建模（STM）模型对象，存储了估计得到的主题模型，包括主题-词分布、文档-主题分布、模型参数，以及复现本研究分析所需的相关元数据。