Bias Benchmark for QA (BBQ)

BBQ是由纽约大学创建的一个手工构建的数据集，旨在评估问答模型中的社会偏见。该数据集包含58,492个示例，覆盖了与美国英语使用环境相关的九个社会维度。每个示例都是由作者根据已验证的社会偏见构建的，旨在测试模型是否系统性地依赖于这些偏见。BBQ的应用领域包括识别模型在不同上下文中的行为可能导致伤害的情况，并探索需要进一步研究和缓解的偏见类型。

BBQ is a manually constructed dataset created by New York University, designed to evaluate social biases in question answering models. It contains 58,492 examples covering nine social dimensions relevant to the context of American English usage. Each example is developed by the dataset's authors based on validated social biases, aiming to test whether models systematically rely on these biases. The application scenarios of BBQ include identifying situations where model behaviors in different contexts may lead to harm, and exploring types of biases that require further research and mitigation.

搜集汇总

背景与挑战

背景概述

Bias Benchmark for QA (BBQ) 是一个由纽约大学创建的手工构建数据集，包含58,492个示例，覆盖九个与美国英语使用环境相关的社会维度，旨在评估问答模型中的社会偏见。该数据集基于已验证的偏见构建，用于测试模型的系统性偏见依赖，并应用于识别模型可能导致伤害的行为和探索偏见缓解方向。