ucinlp/unstereo-eval

---
license: mit
task_categories:
- text-classification
language:
- en
tags:
- bias-evaluation
- language modeling
- gender-bias
pretty_name: USE
size_categories:
- 1K<n<10K
configs:
- config_name: USE-5
data_files:
- split: test
path: "USE-5-no-maxpmi-constraint.csv.gz"
- config_name: USE-10
data_files:
- split: test
path: "USE-10-no-maxpmi-constraint.csv.gz"
- config_name: USE-20
data_files:
- split: test
path: "USE-20-no-maxpmi-constraint.csv.gz"
- config_name: Winobias-all
data_files:
- split: test
path: "Winobias-*-no-maxpmi-constraint.csv.gz"
- config_name: Winogender
data_files:
- split: test
path: "Winogender-no-maxpmi-constraint.csv.gz"
---

# Dataset Card for Dataset Name

We challenge a common observation in prior work considering the gender bias evaluation of large language models (LMs). The observation is that models reinforce stereotypes in the training data by picking up on gendered correlations. In this paper, we challenge this assumption and instead address the question: Do language models still exhibit gender bias in non-stereotypical settings?

To do so, we introduce UnStereoEval (USE), a novel framework tailored for investigating gender bias in stereotype-free scenarios. USE defines a sentence-level score based on pretraining data statistics to determine if the sentence contain minimal word-gender associations. To systematically assess the fairness of popular language models in stereotype-free scenarios, we utilize USE to automatically generate benchmarks without any gender-related language. By leveraging USE's sentence-level score, we also repurpose prior gender bias benchmarks (Winobias and Winogender) for non-stereotypical evaluation.

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->
The dataset constitutes 5 different benchmarks, 3 synthetic and 2 popular gender bias benchmarks, Winobias and Winogender.
The three synthetic benchmarks differ in the template length and were created to be natural-sounding, diverse, gender-invariant and contain words with minimal gender correlations.
The popular Winobias and Winogender constitute manually curated English templates that are gender invariant.
We include the original version of the files but recommend users to restrict the evaluation set to a subset of the sentences that minimizes the gender correlations at a sentence level (`max_gender_pmi`).

- **Curated by:** The authors of the paper
- **Language(s) (NLP):** English
- **License:** MIT 2.0

### Dataset Sources

- **Repository:** https://github.com/ucinlp/unstereo-eval
- **(OpenReview) Paper:** https://openreview.net/forum?id=w1JanwReU6
- **(ArXiv) Paper**: https://arxiv.org/abs/2405.00588
- **Demo:** https://ucinlp.github.io/unstereo-eval/
- **Video:** https://www.youtube.com/watch?v=gmqBoBSYj9U

## Uses

### Direct Use

The dataset should be used to test model capabilities in non-stereotypical sentences and investigate complex model behaviors.

### Out-of-Scope Use

These datasets are not intended for training of current large language models.

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

The benchmark constitutes 5 different datasets, three automatically generated (and prefixed with `USE-`) and 2 well-known gender bias datasets, Winobias and Winogender, that we repurposed.

The provided dataset is not filtered by default. To restrict the evaluation to stereotype free sentences, please consider filtering the dataset by enforcing constraints on the column `max_gender_pmi`.

```python
from datasets import load_dataset

eta = 0.65
use5 = load_dataset("ucinlp/unstereo-eval", "USE-5", split="test")
use5 = use5.filter(lambda example: abs(example["max_gender_pmi"]) <= 0.65)
```

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

Gender bias research has been pivotal in revealing undesirable behaviors in large language models, exposing serious gender stereotypes associated with occupations, and emotions. A key observation in prior work is that models reinforce stereotypes as a consequence of the gendered correlations that are present in the training data. In this paper, we focus on bias where the effect from training data is unclear, and instead address the question: Do language models still exhibit gender bias in non-stereotypical settings?
No such dataset existed, therefore, we created one!

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

Synthetic data, generated by ChatGPT. `USE-*` datasets are created by prompting ChatGPT to generate 5 sentences using `{seed_word}` and one of the male pronouns, *he/his/him*, and another 5 sentences using `{seed_word}` and one of the female pronounts *she/her/her*.
The seed words are randomly selected from a pool of words in PILE for which the gender correlation is minimal.

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

Please check Section 2 of the paper for more details.

#### Who are the source data producers?

- USE-5, USE-10, USE-20 is created by the UCI-NLP using OpenAI's ChatGPT (`gpt-3.5-turbo-1106`, all generations were collected in October 2023).
- Winobias is described in [Zhao et al (2018)](https://aclanthology.org/N18-2003/)
- Winogender is described in [Rudinger et al (2018)](https://aclanthology.org/N18-2002/)

### Annotations

We will briefly describe the procedure. Please refer to the paper for more details on this process.

#### Annotation process

To confirm the quality of the dataset, the authors have manually vetted 250 random samples from each benchmark and ran a small-scale evaluation study with 6 participants (CS researchers) of
various cultural backgrounds.
We conducted a small scale annotation process that involved asking 6 human participants to evaluate 100 randomly selected instances from each generated benchmarks.
We found that on average 97% of examples are considered neutral and that 98% of the examples are considered neutral by at least 5 annotators.

#### Who are the annotators?

The first three authors of the paper manually annotated 250 random samples from each benchmark.
The small-scale evaluation was conducted by asking graduate student researchers across Machine Learning labs at UCI.

#### Personal and Sensitive Information

No personal or sensitive information is present in the current dataset.

## Bias, Risks, and Limitations

One limitation of the current study is the focus on binary gender bias and the assessment of fairness solely using the English pronouns “she” and “he”. The extent to which these findings apply
to non-binary gender identities or to other demographic groups (e.g., racism, cultural) remains an
open question. Future research could investigate the applicability of our findings across different groups and languages, as well as expand the gender co-occurrence definition to include multiple
gendered expressions.

Another limitation of our work is the use of a single model to construct nonstereotypical benchmarks, which may limit the diversity of the dataset and introduce model-specific
artifacts. To confirm the quality of the dataset, the authors have manually vetted 250 random samples from each benchmark and ran a small-scale evaluation study with 6 participants (CS researchers) of
various cultural backgrounds. The participants were asked to evaluate 100 randomly selected instances from each generated benchmarks.
We found that on average 97% of examples are considered neutral and that 98% of the examples are considered neutral by at least 5 annotators.
We encourage future research to run more comprehensive analysis of the quality and potential artifacts introduced by constructing benchmarks with different models,
such as Claude or Llama-2 (Anthropic, 2024; Touvron et al., 2023).

## Citation

**BibTeX:**

```
@inproceedings{belem2024-unstereoeval,
title={Are Models Biased on Text without Gender-related Language?},
author={Catarina G Bel{\'e}m and Preethi Seshadri and Yasaman Razeghi and Sameer Singh},
month={May},
year={2024},
booktitle={The Twelfth International Conference on Learning Representations},
url={https://openreview.net/forum?id=w1JanwReU6}
}
```

**APA:**

```
Belém, C. G., Seshadri, P., Razeghi, Y., & Singh, S. (2024, May). Are Models Biased on Text without Gender-related Language? The Twelfth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=w1JanwReU6
```

## Glossary

In this section, we describe in greater detail a few of the conepts used in the dataset.
Please refer to the paper for more comprehensive information.

- **Templates**: We define a template to be a sentence containing a few variables (dubbed placeholders).
By replacing the pronouns with the possible defined values, we obtain different variants of the same template.
One example of a template is *"We appreciate that {pronoun}'s here."*, where we use the notation `{...}` to denote placeholders, i.e., a variable whose value will be replaced to generate a sentence.

- **Sentence pair**: In UnStereoEval we create sentence pairs from individual templates by replacing the placeholders with binary gendered pronouns. Specifically, we will use the following mapping:
- `{pronoun}` is mapped to "he" or "she";
- `{pronoun1}`is mapped to "his" or "her";
- `{pronoun2}` is mapped to "him" or "her".

- **Non stereotypical templates**: We define non-stereotypical templates as being templates that when replaced by each of the gender pronouns lead (1) fluent, (2) semantically and grammatically correct sentences, and (3) its words are minimally correlated with gender. This gender correlation is determined using the pretraining data co-occurrence statistics between the words in a sentence and the pronouns "he" and "she".

- **Preference**: For a given template, we say that a language model has no gender preference if the log likelihood assigned by the model to the individual sentences is approximately the same. To be more precise, we say the model manifests no bias for the specific template if the likelihood ratio between the two gendered sentences is less than or equal to 1.65x.

## Dataset Card Authors

Catarina Belem

## Dataset Card Contact

[Catarina Belem](mailto:[e-mail])

原始信息汇总

数据集概述

数据集名称

• 名称: UnStereoEval (USE)

数据集描述

• 目的: 用于评估大型语言模型在非典型性别偏见场景下的表现。
• 构成: 包含5个不同的基准，其中3个为合成基准（USE-5, USE-10, USE-20），2个为流行的性别偏见基准（Winobias和Winogender）。
• 特点: 所有基准均设计为性别无关，包含的句子具有最小的词-性别关联。

数据集详细信息

数据集描述

• 内容: 由5个不同的基准组成，包括3个合成基准和2个流行的性别偏见基准。
• 语言: 英语
• 许可证: MIT 2.0

数据集来源

• 创建者: 论文作者
• 数据收集与处理: 使用ChatGPT生成合成数据，确保性别无关性。
• 注释过程: 通过手动验证和一个小规模的评估研究来确认数据集质量。

数据集结构

• 配置: 包含多个配置，每个配置对应不同的数据文件和测试集分割。
• 数据文件: 每个配置下有特定的数据文件，如USE-5-no-maxpmi-constraint.csv.gz等。

数据集使用

• 直接用途: 用于测试模型在非典型句子中的能力，并研究复杂的模型行为。
• 超出范围的用途: 不建议用于当前大型语言模型的训练。

数据集创建

• 动机: 为了研究语言模型在非典型性别偏见设置中是否仍显示性别偏见。
• 源数据: 合成数据由ChatGPT生成，确保性别无关性。

偏见、风险和限制

• 限制: 当前研究主要关注二元性别偏见，且仅使用英语代词“她”和“他”进行评估。
• 未来研究方向: 建议未来研究扩展到非二元性别身份和其他语言，以及使用多个模型构建非典型基准。

引用信息

• BibTeX: 提供详细的引用信息。

术语解释

• 模板: 包含变量的句子，通过替换代词生成不同的句子变体。
• 非典型模板: 当替换为性别代词时，生成流畅、语义和语法正确的句子，且词与性别关联最小。

数据集卡作者和联系信息

• 作者: Catarina Belem
• 联系: [e-mail]

搜集汇总

数据集介绍

构建方式

在自然语言处理领域，性别偏见评估通常聚焦于模型对训练数据中性别关联的强化现象。UnStereoEval（USE）数据集的构建旨在突破这一局限，通过创新框架在无性别刻板印象的语境下探究语言模型的偏见表现。该数据集包含三个自动生成的基准（USE-5、USE-10、USE-20）以及两个经过改造的经典性别偏见基准（Winobias与Winogender）。自动生成部分借助ChatGPT模型，以性别关联度极低的种子词为基础，通过模板化提示生成包含男女性别代词的句子对，确保内容自然多样且性别中立。构建过程中，研究者基于预训练数据的统计量计算句子级性别关联分数，并辅以人工标注验证，最终筛选出性别关联度最低的句子，形成非刻板印象评估基准。

特点

USE数据集的显著特征在于其专注于非刻板印象语境下的性别偏见评估，填补了该领域的研究空白。数据集通过量化指标（如max_gender_pmi）精确衡量句子与性别的关联强度，使评估更具客观性与可重复性。其内容涵盖不同长度的模板结构，既包含自动生成的合成数据，也整合了经典基准的改造版本，提供了多维度、层次化的评估场景。数据集中所有句子均经过严格筛选，确保词汇与性别的相关性最小化，同时通过人工验证确认了高达97%的中立性，有效降低了数据偏差对评估结果的干扰。这种设计使得USE能够深入揭示语言模型在无显性性别线索语境中的潜在偏见行为。

使用方法

使用USE数据集时，研究者需首先通过Hugging Face的datasets库加载相应配置，例如USE-5或Winobias-all。为聚焦非刻板印象评估，建议对数据施加过滤条件，依据max_gender_pmi列的绝对值设定阈值（如0.65），以剔除性别关联较强的句子。该数据集主要用于测试语言模型在非刻板印象句子中的表现，通过对比模型对男女性别代词句子的似然值差异，计算偏见程度。典型应用包括分析模型在无性别关联词汇语境下的公平性，或探究偏见与训练数据关联的复杂性。需要注意的是，数据集仅适用于评估与研究，不应用于模型训练，且当前版本仅支持英语及二元性别评估。

背景与挑战

背景概述

在自然语言处理领域，性别偏见评估一直是模型公平性研究的关键议题。2024年，加州大学欧文分校自然语言处理团队（UCI-NLP）的Catarina G. Belém等人提出了UnStereoEval（USE）数据集，旨在探究大型语言模型在非刻板印象语境下的性别偏见表现。该数据集通过构建性别关联度极低的句子模板，挑战了传统偏见评估中模型仅强化训练数据性别关联的假设，为深入理解模型内在偏见机制提供了新颖的评估框架。

当前挑战

USE数据集致力于解决非刻板印象语境中性别偏见检测的挑战，其核心在于如何精准界定并生成完全中性的语言环境，以隔离训练数据中性别关联的干扰。在构建过程中，研究团队面临合成数据质量控制的难题，需确保自动生成的句子在保持自然性与多样性的同时，最小化词汇与性别的统计关联。此外，数据集的局限性体现在对二元性别及英语代词的聚焦，未能涵盖非二元性别身份或多语言场景，且依赖单一模型生成可能引入模型特定偏差，这些因素均对评估的普适性与鲁棒性构成挑战。

常用场景

经典使用场景

在自然语言处理领域，性别偏见评估一直是模型公平性研究的核心议题。UnStereoEval（USE）数据集通过构建无性别关联的文本场景，为评估大型语言模型在非刻板印象语境下的偏见表现提供了标准化基准。该数据集包含自动生成的合成语句及重构的经典偏见评估集，研究者可借助其句子级性别关联度评分，系统性地检验模型在性别中立语言环境中的行为模式，从而揭示模型内部潜在的隐性偏见机制。

解决学术问题

传统性别偏见研究多聚焦于训练数据中显性的性别关联，而USE数据集突破了这一局限，致力于探究语言模型在无性别关联语境中是否仍存在系统性偏差。该数据集通过量化句子层面的性别共现统计量，解决了非刻板印象场景下偏见检测的标准化难题，为理解模型偏见与训练数据关联性的边界提供了实证基础，推动了公平性评估从表象关联向深层机制的研究范式转变。

衍生相关工作

基于USE数据集构建的评估框架，学界衍生出多维度偏见检测研究体系。部分研究扩展了其方法论至跨文化偏见评估，通过构建多语言无偏见语料库检验模型的文化适应性；另有工作将其与因果推断方法结合，开发出可解释的偏见溯源工具。这些衍生研究共同推动了从二进制性别评估向多元身份维度、从静态检测向动态干预的公平性研究演进。