Maltese crowS-pairs dataset

Warning: This dataset contains explicit statements of offensive stereotypes which may be upsetting.The study of bias, fairness and social impact in Natural Language Processing (NLP) lacks resources in languages other than English. Our objective is to support the evaluation of bias in language models in a multilingual setting. We use stereotypes across nine types of biases to build a corpus containing contrasting sentence pairs, one sentence that presents a stereotype concerning an underadvantaged group and another minimally changed sentence, concerning a matching advantaged group.In total, we produced 11,139 new sentence pairs that cover stereotypes dealing with nine types of biases in seven cultural contexts. We use the final resource for the evaluation of relevant monolingual and multilingual masked language models.This file contains the sentence pairs localised to the Maltese context in the Maltese language.Other languages are available here: https://gitlab.inria.fr/corpus4ethics/multilingualcrowspairsThe paper describing this work is available here: https://www.um.edu.mt/library/oar/handle/123456789/121722https://aclanthology.org/2024.lrec-main.1545/To use this dataset, please use the following citation:Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, et al.. 2024. Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17764–17769, Torino, Italia. ELRA and ICCL.

警告：本数据集包含明确的冒犯性刻板印象表述，可能会引起不适。
自然语言处理（Natural Language Processing, NLP）领域中关于偏见、公平性与社会影响的研究，在英语以外的语言中缺乏相关资源。本研究旨在支持多语言场景下语言模型的偏见评估。
我们基于九类偏见的刻板印象构建了包含对比句对的语料库：一句针对弱势群体的刻板印象表述，另一句仅做极小改动，对应匹配的优势群体。
总计我们生成了11139条全新的句对，覆盖七种文化语境下九类偏见相关的刻板印象。本最终数据集将用于相关单语言与多语言掩码语言模型（Masked Language Model）的偏见评估。
本文件包含适配马耳他文化语境的马耳他语句对。
其他语言版本可通过以下链接获取：https://gitlab.inria.fr/corpus4ethics/multilingualcrowspairs
本研究的相关论文可通过以下链接获取：https://www.um.edu.mt/library/oar/handle/123456789/121722 以及 https://aclanthology.org/2024.lrec-main.1545/
若需使用本数据集，请引用以下文献：
Karen Fort、Laura Alonso Alemany、Luciana Benotti、Julien Bezançon、Claudia Borg、Marthese Borg、Yongjian Chen、Fanny Ducel、Yoann Dupont、Guido Ivetta、Zhijian Li、Margot Mieskes、Marco Naguib、Yuyan Qian、Matteo Radaelli、Wolfgang S. Schmeisser-Nieto、Emma Raimundo Schulz、Thiziri Saci、Sarah Saidi等. 2024. Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17764–17769, Torino, Italia. ELRA and ICCL.