GeNTE

------------------------------------------------------------------------------------------------------------------- # 🚨 GeNTE has been superseded by [**mGeNTE**](https://huggingface.co/datasets/FBK-MT/mGeNTE), a new multilingual release of the corpus with additional annotations. ------------------------------------------------------------------------------------------------------------------- # Dataset Card for GeNTE **Homepage:** https://mt.fbk.eu/gente/ ## Dataset Summary GeNTE (**Ge**nder-**N**eutral **T**ranslation **E**valuation) is a natural, bilingual corpus designed to benchmark the ability of machine translation systems to generate gender-neutral translations. Built from European Parliament speeches, GeNTE comprises a subset of the English-Italian portion of the [Europarl corpus](https://www.statmt.org/europarl/archives.html). GeNTE comprises 1500 parallel sentences, which are enriched with manual annotations and feature a balanced distribution of translation phenomena that either entail i) a gender-neutral translation (`Set-N`), or ii) a gendered translation in the target language (`Set-G`). ### Supported Tasks and Languages **Machine Translation** GeNTE supports cross-lingual (en-it) and intra-lingual (it-it) gender inclusive translation tasks. To evaluate with GeNTE you can use the gender-neutrality classifier available on Hugging Face at [FBK-MT/GeNTE-evaluator](https://huggingface.co/FBK-MT/GeNTE-evaluator). For additional details on evaluation with GeNTE, please refer to the paper [*Hi Guys* or *Hi Folks?* Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus](https://aclanthology.org/2023.emnlp-main.873/). Code and data for the classifier are available at [fbk-NEUTR-evAL](https://github.com/hlt-mt/fbk-NEUTR-evAL/blob/main/solutions/GeNTE.md). This version of GeNTE has been used in the following papers: * [_Hi Guys_ or _Hi Folks_? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus](https://aclanthology.org/2023.emnlp-main.873/) (Piergentili et al., 2023 – EMNLP) * [A Prompt Response to the Demand for Automatic Gender-Neutral Translation](https://aclanthology.org/2024.eacl-short.23/) (Savoldi et al., 2024 – EACL) * [ItaEval and TweetyIta: A New Extensive Benchmark and Efficiency-First Language Model for Italian](https://ceur-ws.org/Vol-3878/6_main_long.pdf) (Attanasio et al., 2024 – CLiC-it) * [ItaEval: A CALAMITA Challenge](https://ceur-ws.org/Vol-3878/117_calamita_long.pdf) (Attanasio et al., 2024 – CALAMITA) * [GFG - Gender-Fair Generation: A CALAMITA Challenge](https://ceur-ws.org/Vol-3878/122_calamita_long.pdf) (Frenda et al., 2024 – CALAMITA) If you plan to use GeNTE in your work, please refer to the updated corpus available at [FBK-MT/mGeNTE](https://huggingface.co/datasets/FBK-MT/mGeNTE). ## Dataset Structure ### Data Instances The dataset consists of two configuration types (`main` and `common`) corresponding to the files: - **`GeNTE.tsv`:** The complete GeNTE corpus and its set annotations - **`GeNTE_common.tsv`:** Subset of the GeNTE corpus that comprises 3 alternative gender-neutral reference translations ### Data Fields **`GeNTE.tsv`** is organized into 8 tab-separated columns as follows: - ID: The unique GeNTE ID. - Europarl_ID: The original sentence ID from Europarl's common-test-set 2. - SET: Indicates whether the entry belongs to the Set-G or the Set-N subportion of the corpus. - SRC: The English source sentence. - REF-G: The gendered Italian reference translation. - REF-N: The gender-neutral Italian reference, produced by a professional translator. - COMMON: Indicates whether the entry is part of GeNTE common-set (yes/no). - GENDER: For entries belonging to the Set-G, indicates if the the entry is Feminine or Masculine (F/M). For entries of the common set, REF-N provides the gender-neutral Italian reference translation n. 2. **`GeNTE-common.tsv`** comprises 200 entries organized into 9 tab-separated columns as follows: - ID: The unique GeNTE ID. - Europarl_ID: The original sentence ID from Europarl's common-test-set 2. - SET: Indicates whether the entry belongs to the Set-G or the Set-N subportion of the corpus. - SRC: The English source sentence. - REF-G: The gendered Italian reference translation. - REF-N1: The gender-neutral Italian reference produced by Translator 1. - REF-N2: The gender-neutral Italian reference produced by Translator 2. - REF-N3: The gender-neutral Italian reference produced by Translator 3. - GENDER: For entries belonging to the Set-G, indicates if the the entry is Feminine or Masculine (F/M). ## Dataset Creation Refer to the original [paper](https://aclanthology.org/2023.emnlp-main.873/) for full details on dataset creation. ### Curation Rationale GeNTE is designed to evaluate models’ ability to perform gender-neutral translations under desirable circumstances. In fact, when referents’ gender is unknown or irrelevant, undue gender inferences should not be made and translation should be neutral. Instead, when a referent’s gender is relevant and known, MT should not over-generalize to neutral translations. The corpus hence consists parallel sentences with mentions to human referents that equally represent two translation scenarios: - `Set-N`: featuring gender-ambiguous source sentences that require to be neutrally rendered in translation; - `Set-G`: featuring gender-unambiguous source sentences, which shall be properly rendered with gendered (masculine or feminine) forms in translation. ### Source Data The dataset contains text data extracted and edited from the Europarl Corpus ([common test set 2](https://www.statmt.org/europarl/archives.html)), and all rights of the data belong to the European Union and/or respective copyright holders. Please refer to Europarl “[Terms of Use](https://www.statmt.org/europarl/archives.html)” for details. ### Annotations For each sentence pair extracted from Europarl (src, it-ref),GeNTE includes an additional Italian reference, which differs from the original one only in that it refers to the human entities with neutral expressions. The neutral reference translation were created by professionals based on the following [guidelines](https://drive.google.com/file/d/1TvV6NQoXiPHNSUHYlf4NFhef1_PKncF6/view?usp=sharing). ### Dataset Curators The authors of GeNTE are the dataset curators. - Beatrice Savoldi (FBK): [e-mail] - Luisa Bentivogli (FBK): [e-mail] - Andrea Piergentili (FBK): [e-mail] ### Licensing Information The GeNTE corpus is released under a Creative Commons Attribution 4.0 International license (CC BY 4.0). ## Citation ```bibtex @inproceedings{piergentili-etal-2023-hi, title = "Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the {G}e{NTE} Corpus", author = "Piergentili, Andrea and Savoldi, Beatrice and Fucci, Dennis and Negri, Matteo and Bentivogli, Luisa", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.emnlp-main.873", doi = "10.18653/v1/2023.emnlp-main.873", pages = "14124--14140" } ``` ## Contributions Thanks to [@BSavoldi](https://huggingface.co/BSavoldi) for adding this dataset.