Dataset for "Beyond Viewpoint Affinity: Measuring Political Bias in LLMs as a Failure of Epistemic Consistency"

Experimental outputs for manuscript "Beyond Viewpoint Affinity: Measuring Political Bias in LLMs as a Failure of Epistemic Consistency" Political bias in large language models is often measured by treating responses to political questionnaires as evidence of ideological preference. This approach can conflate bias with differences in evidential support: asymmetric viewpoint preferences are not necessarily biased if one side is better supported by the available evidence. We explore a different facet of political bias: failures of epistemic consistency, in which models apply different evidentiary or evaluative standards to substantively equivalent items that carry different political cues. We operationalize this definition in two sets of experimental tasks. In person-attribution tasks, models evaluate nonpolitical items—such as mathematical proofs, logical arguments, or artistic artifacts---while only the political identity of the person associated with the item changes. In politicized-context tasks, models judge politically framed items, such as research designs, news articles, or policy outcomes, while partisan sources, group identities, or the left–right valence of the substantive outcome change. Across thirty LLMs, person-attribution tasks yield only modest left-leaning asymmetry—and this bias shrinks toward zero in more capable models. By contrast, politicized-context tasks yield large left-leaning asymmetries that do not shrink with model capability. Debiasing interventions---such as increasing reasoning effort, system prompts that emphasize epistemic rigor, and explicit instructions to ignore specific political cues—mitigate bias, but only partially. These results suggest that current LLMs show little bias based on an individual’s political identity when evaluating nonpolitical or politically orthogonal items. However, on politicized-context tasks, LLMs are persistently biased by task-irrelevant political cues, even when the relevant evidence is objectively assessable or held constant across political variants. Note: A previous version of the experimental_results archive contained partial and incomplete outputs for the legacy OpenAI model gpt-4-0613 from 2023. These results were not included in the analyses reported in the paper. The run was stopped partway through after we observed that this older model was incurring excessive API costs. As of April 2026, the input/output costs per 1M tokens of gpt-4-0613 were 30\$ and 60\$ respectively (between one and several orders of magnitude higher than newer OpenAI models used in the study, see: https://developers.openai.com/api/docs/models/gpt-4). Completing the full benchmark for gpt-4-0613 would have caused the project to exceed its compute budget, so we created a new version of the experimental outputs archive with those incomplete files for gpt-4-0613 removed. In the code repository (https://anonymous.4open.science/r/llms_epistemic_consistency-D133/README.md) associated with this experimental output there is the following bug for the Debiasing interventions using system prompts ("centrist" and "epistemically rigorous") experiment, which does not affect the experimental results but affects one field of the meta information saved to the CSV files In vpei/epistemic_consistency/experiment_types.py, all six run_model functions build a payload dict before theAPI call: payload = { "system_prompt": system_prompt, # ← saves the BASE prompt only ...}messages = [{"role": "system", "content": system_prompt}, ...]tasks.append((payload, make_llm_request_async(model_name, messages, ...))) Then inside make_llm_request_async → prepare_request_params() (llm_requests_v3.py:309–314), thesystem_prompt_prefix (EPISTEMICALLY_RIGOROUS_LLM_SYSTEM_PROMPT) is prepended to the system message after thepayload is already constructed. So: - LLM API receives: EPISTEMICALLY_RIGOROUS_LLM_SYSTEM_PROMPT + '\n' + system_prompt - CSV saves in system_prompt column: just bare system_prompt — the prefix is silently dropped (the bug)- Downstream analysis of "model_response" column remains correct This bug is circumscribed to the debiasing intervention using system prompts.论文《超越视角亲和性：将大语言模型中的政治偏见衡量为认知一致性失效》的实验输出 政治偏见的传统衡量方式通常将大语言模型（Large Language Model，LLM）对政治问卷的回复作为意识形态偏好的证据，但该方法易将偏见与证据支持的差异相混淆：若某一立场有更充分的可用证据支撑，那么不对称的视角偏好未必属于偏见。本文聚焦政治偏见的另一维度：认知一致性失效，即模型对实质等价但携带不同政治暗示的同类项目，采用了不同的证据标准或评价尺度。我们通过两组实验任务对该定义进行操作化：其一为人物归因任务，模型需评价非政治项目（如数学证明、逻辑论证或艺术作品），仅关联项目的人物政治身份发生变化；其二为政治化语境任务，模型需评判带有政治框架的项目（如研究设计、新闻报道或政策结果），而党派来源、群体身份或实质结果的左右倾向性会发生变化。 针对30个大语言模型的实验结果显示，人物归因任务仅产生温和的左倾不对称偏见，且在能力更强的模型中该偏见会趋近于零。与之形成鲜明对比的是，政治化语境任务会产生显著的左倾不对称偏见，且该偏见不会随模型能力提升而减弱。去偏干预措施——包括增加推理思考环节、强调认知严谨性的系统提示，以及明确要求忽略特定政治暗示的指令——仅能部分缓解偏见。上述结果表明，当前的大语言模型在评价非政治或与政治无关的项目时，几乎不会因个体政治身份产生偏见。然而在政治化语境任务中，即便相关证据可客观评估或在不同政治变体中保持一致，大语言模型仍会受与任务无关的政治暗示持续影响而产生偏见。 注意：此前版本的实验结果归档包含了2023年旧版OpenAI模型gpt-4-0613的部分不完整输出，这些结果并未纳入论文报告的分析中。在发现该旧模型产生过高的API调用成本后，我们中途终止了运行。截至2026年4月，gpt-4-0613每百万Token的输入/输出成本分别为30美元和60美元（比本研究中使用的新版OpenAI模型高出1至数个数量级，详见：https://developers.openai.com/api/docs/models/gpt-4）。若要完成gpt-4-0613的完整基准测试，将导致项目超出计算预算，因此我们更新了实验输出归档，移除了其中关于gpt-4-0613的不完整文件。 在与该实验输出关联的代码仓库（https://anonymous.4open.science/r/llms_epistemic_consistency-D133/README.md）中，针对使用系统提示（“中间派”与“认知严谨”）的去偏干预实验存在一处bug，该bug不影响实验结果，但会影响保存至CSV文件的元信息中的一个字段： 在vpei/epistemic_consistency/experiment_types.py中，所有六个run_model函数均会在API调用前构建负载字典： payload = { "system_prompt": system_prompt, # ← 仅保存基础提示 ...} messages = [{"role": "system", "content": system_prompt}, ...] tasks.append((payload, make_llm_request_async(model_name, messages, ...))) 随后在make_llm_request_async → prepare_request_params()（llm_requests_v3.py:309–314）中，系统提示前缀（EPISTEMICALLY_RIGOROUS_LLM_SYSTEM_PROMPT）会在负载字典构建完成后被添加至系统消息的开头。因此： - LLM API实际接收的内容为：EPISTEMICALLY_RIGOROUS_LLM_SYSTEM_PROMPT + ' ' + system_prompt - CSV文件的system_prompt列仅保存了原始的基础system_prompt，前缀被静默移除（该bug） - 对“model_response”列的下游分析仍保持正确 该bug仅局限于使用系统提示的去偏干预实验。