Bias, Randomness, and Blind-Faith: Large Language Model Code Generation and Security Analysis

Testing bias in code generation and security in large language models (LLMs) ChatGPT, Claude, and Gemini. This data is accompanying a paper submitted to USENIX '25. In experimentation, three trials were completed, each testing five different categories. The file Trial Charts.zip includes the trial model versions, final results from each category and test, as well as results from manual analysis. Each bias has its own .csv (Sex, Age, Race & Ethnicity, Experience, Special Circumstances) for every trial's results and another file with the first letter of the bias + Overall (i.e. A Overall for Age, SC Overall for Special Circumstances) for overall results of the category. The files labeled Manual Analysis (or Manual A.) highlight some of the differences between the biased code and the control code. It also includes notable outputs from each bias. The Trial Results.zip includes all of the raw data from testing the biases. The data was collected in OneNote. The files "Trial [1-3]" include all outputs from all categories. The files "Trial [1-3] Retests" include any non-control retests. The files "[1-3] Control Retests" give the results of retesting the control for all three trials. The files "[1-3] A and B Tests" show the results from resting with the new labels "a" and "b".本数据集配套一篇提交至USENIX '25的学术论文，旨在测试大语言模型（Large Language Models，LLMs）ChatGPT、Claude及Gemini的代码生成偏倚与安全性表现。实验共开展三轮测试，每轮均覆盖五个不同测试类别。 压缩包"Trial Charts.zip"收录了本次测试所用的模型版本、各分类与测试的最终结果，以及人工分析结果。针对每轮测试，各类偏倚（性别、年龄、种族与族裔、经验、特殊场景）均配有独立的.csv文件；另有一类文件以偏倚首字母加"Overall"命名（例如年龄偏倚对应A Overall、特殊场景偏倚对应SC Overall），用于汇总该类别的整体测试结果。标注为"Manual Analysis（或Manual A.）"的文件展示了存在偏倚的代码与对照代码间的部分差异，同时收录了各类偏倚下的典型输出样本。 压缩包"Trial Results.zip"收录了所有偏倚测试的原始数据，所有数据均采集自OneNote。其中"Trial [1-3]"系列文件收录了所有测试类别的全部输出结果；"Trial [1-3] Retests"系列文件收录了非对照项的复测结果；"[1-3] Control Retests"系列文件给出了三轮测试中对照项的复测结果；"[1-3] A and B Tests"系列文件展示了使用新标签"a"与"b"开展测试的结果。