quantibias
收藏资源简介：
QuantiBias 是一个用于评估大型语言模型（LLM）后训练量化所引发偏见的基准测试协议，而非传统静态数据集。其核心目标是揭示量化压缩可能导致的特定故障模式：在保持短格式安全行为（如直接拒绝、过度拒绝、多项选择偏见得分）基本不变的同时，使模型在开放式生成中主动表达的偏见显著上升。该协议通过可复现的测试框架操作，不直接分发数据，而是提供工具从八个已发布的上游多语言偏见基准（涵盖英语、阿拉伯语、德语、西班牙语、法语、日语、俄语和中文）中确定性地采样构建探测集。测量流程包括：1) 重建精确的评估探测集；2) 在推理开启或关闭模式下，让模型生成针对刻板印象提示的答案；3) 使用独立于被测模型的LLM作为评判员，依据评分标准（包括刻板印象倾向、内容严重性等级和偏见延续）对生成内容评分；4) 将评分聚合成量化偏见指标，并计算“选择性差距”——即短格式控制指标平稳而开放式刻板印象认可率高的现象。所有结果索引到模型权重的实测有效比特/权重值。该协议旨在用于模型偏见审计和安全研究，特别评估量化部署版本相较于全精度版本是否引入了额外开放式生成偏见。
QuantiBias is a benchmark protocol, not a traditional static dataset, for evaluating biases induced by post-training quantization of large language models (LLMs). Its core objective is to reveal a specific, under-audited failure mode: quantized compression may leave the short-form safety behaviors of the model—such as direct refusal, over-refusal, and multiple-choice bias scores—nearly unchanged, while simultaneously causing a significant increase in the actively expressed biases during open-ended generation. This protocol operates via a reproducible test framework that does not directly distribute data, but instead provides tools to deterministically construct probe sets by sampling from eight published upstream multilingual bias benchmarks covering English, Arabic, German, Spanish, French, Japanese, Russian, and Chinese. The core measurement workflow includes: 1) Reconstructing the exact evaluation probe sets from the upstream sources; 2) Having the target model generate responses to stereotype prompts with inference mode either enabled or disabled; 3) Using an LLM independent of the target model family as a judge to score the generated content based on defined scoring criteria, including stereotype tendency, content severity level, and bias perpetuation; 4) Aggregating the scoring results into quantifiable bias metrics and calculating the "selectivity gap"—the phenomenon where short-form control metrics remain stable while the open-ended stereotype endorsement rate is high. All results are indexed to the measured effective bits per weight value of the model weights. This protocol is intended for model bias auditing and safety research, and is particularly suitable for evaluating whether quantized deployed versions introduce additional open-ended generation biases compared to their full-precision evaluation counterparts.
