quantiles/crows_pairs

CrowS-Pairs是一个用于评估掩码语言模型中社会偏见的挑战数据集。它包含1508个英语句子对，每个句子对包括一个更具偏见的句子（sent_more）和一个较少偏见的句子（sent_less），用于衡量模型在生成或评分时是否表现出社会偏见。数据集覆盖了多种偏见类型，如种族、性别、社会经济地位、残疾、国籍、性取向、外貌、宗教和年龄。每个示例都标注了偏见类型（bias_type）和刻板/反刻板标签（stereo_antistereo），并提供了匿名作者和注释者信息。数据集的创建基于ROCStories语料库和MNLI的虚构部分，旨在帮助研究人员检测和缓解语言模型中的偏见问题。

CrowS-Pairs is a challenge dataset designed to measure social biases in masked language models. It consists of 1508 English sentence pairs, each containing a more biased sentence (sent_more) and a less biased sentence (sent_less), used to evaluate whether models exhibit social biases in generation or scoring tasks. The dataset covers multiple bias types, including race-color, gender, socioeconomic status, disability, nationality, sexual orientation, physical appearance, religion, and age. Each example is annotated with bias type (bias_type) and stereo/antistereo labels (stereo_antistereo), and includes anonymous writer and annotator information. The dataset is created using prompts from the ROCStories corpus and the fiction part of MNLI, aiming to assist researchers in detecting and mitigating bias in language models.