multilingual-crows-pairs/multilingual-crows-pairs

---
dataset_info:
features:
- name: sent_more
dtype: string
- name: sent_less
dtype: string
- name: stereo_antistereo
dtype: string
- name: bias_type
dtype: string
- name: id
dtype: int64
splits:
- name: es_AR
num_bytes: 275173
num_examples: 1509
download_size: 165548
dataset_size: 275173
configs:
- config_name: default
data_files:
- split: es_AR
path: data/es_AR-*
license: cc-by-sa-4.0
language:
- es
---

## Citation

```
@inproceedings{fort-etal-2024-stereotypical,
title = "Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts",
author = "Fort, Karen and
Alonso Alemany, Laura and
Benotti, Luciana and
Bezan{\c{c}}on, Julien and
Borg, Claudia and
Borg, Marthese and
Chen, Yongjian and
Ducel, Fanny and
Dupont, Yoann and
Ivetta, Guido and
Li, Zhijian and
Mieskes, Margot and
Naguib, Marco and
Qian, Yuyan and
Radaelli, Matteo and
Schmeisser-Nieto, Wolfgang S. and
Raimundo Schulz, Emma and
Saci, Thiziri and
Saidi, Sarah and
Torroba Marchante, Javier and
Xie, Shilin and
Zanotto, Sergio E. and
N{\'e}v{\'e}ol, Aur{\'e}lie",
editor = "Calzolari, Nicoletta and
Kan, Min-Yen and
Hoste, Veronique and
Lenci, Alessandro and
Sakti, Sakriani and
Xue, Nianwen",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.1545",
pages = "17764--17769",
abstract = "Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting The study of bias, fairness and social impact in Natural Language Processing (NLP) lacks resources in languages other than English. Our objective is to support the evaluation of bias in language models in a multilingual setting. We use stereotypes across nine types of biases to build a corpus containing contrasting sentence pairs, one sentence that presents a stereotype concerning an underadvantaged group and another minimally changed sentence, concerning a matching advantaged group. We build on the French CrowS-Pairs corpus and guidelines to provide translations of the existing material into seven additional languages. In total, we produce 11,139 new sentence pairs that cover stereotypes dealing with nine types of biases in seven cultural contexts. We use the final resource for the evaluation of relevant monolingual and multilingual masked language models. We find that language models in all languages favor sentences that express stereotypes in most bias categories. The process of creating a resource that covers a wide range of language types and cultural settings highlights the difficulty of bias evaluation, in particular comparability across languages and contexts.",
}
```

数据集信息：
特征字段：
- 名称：偏向刻板印象语句（sent_more），数据类型：字符串
- 名称：弱化刻板印象语句（sent_less），数据类型：字符串
- 名称：刻板/反刻板标注（stereo_antistereo），数据类型：字符串
- 名称：偏见类型（bias_type），数据类型：字符串
- 名称：编号（id），数据类型：整数
划分集：
- 名称：阿根廷西班牙语（es_AR），字节数：275173，样本量：1509
下载大小：165548
数据集总大小：275173
配置项：
- 配置名称：默认（default），数据文件：
- 划分集：阿根廷西班牙语（es_AR），路径：data/es_AR-*
许可证：cc-by-sa-4.0
语言：西班牙语（es）

## 参考文献

@inproceedings{fort-etal-2024-stereotypical,
title = "你的刻板印象适用范围因人而异：多语言与文化语境下评估偏见的实际挑战",
author = "Fort, Karen 及
Alonso Alemany, Laura 及
Benotti, Luciana 及
Bezançon, Julien 及
Borg, Claudia 及
Borg, Marthese 及
Chen, Yongjian 及
Ducel, Fanny 及
Dupont, Yoann 及
Ivetta, Guido 及
Li, Zhijian 及
Mieskes, Margot 及
Naguib, Marco 及
Qian, Yuyan 及
Radaelli, Matteo 及
Schmeisser-Nieto, Wolfgang S. 及
Raimundo Schulz, Emma 及
Saci, Thiziri 及
Saidi, Sarah 及
Torroba Marchante, Javier 及
Xie, Shilin 及
Zanotto, Sergio E. 及
Névéol, Aurélie",
editor = "Calzolari, Nicoletta 及
Kan, Min-Yen 及
Hoste, Veronique 及
Lenci, Alessandro 及
Sakti, Sakriani 及
Xue, Nianwen",
booktitle = "2024年国际计算语言学、语言资源与评估联合会议（LREC-COLING 2024）论文集",
month = 五月,
year = "2024",
address = "意大利都灵",
publisher = "ELRA与ICCL",
url = "https://aclanthology.org/2024.lrec-main.1545",
pages = "17764--17769",
abstract = "警告：本文包含露骨的冒犯性刻板表述，可能引起不适。自然语言处理（Natural Language Processing，NLP）领域关于偏见、公平性与社会影响的研究，在英语以外的语言中缺乏相关资源。本研究旨在支撑多语言场景下的语言模型偏见评估工作。我们基于九类偏见场景下的刻板印象，构建包含对立语句对的语料库：其中一句针对弱势群体呈现刻板印象，另一句仅做极小改动，对应匹配的优势群体。我们以法语CrowS-Pairs语料库及编写指南为基础，将现有材料翻译为另外七种语言。最终我们共生成11139条全新语句对，覆盖七种文化语境下的九类偏见相关刻板印象。我们使用该最终资源对相关单语与多语掩码语言模型（masked language model）开展评估，发现所有语言的语言模型在多数偏见类别中均倾向于输出表达刻板印象的语句。构建覆盖广泛语言类型与文化场景的资源这一过程，凸显了偏见评估的难点，尤其是跨语言与跨语境的可比性问题。",
}