demelin/wino_x

---
annotations_creators:
- no-annotation
language:
- en
- de
- fr
- ru
language_creators:
- machine-generated
- expert-generated
license:
- mit
multilinguality:
- multilingual
- translation
pretty_name: Wino-X
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- translation
- coreference resolution
- commonsense reasoning
task_ids:
- multiple-choice-qa
- language-modeling
---

# Dataset Card for Wino-X

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-instances)
- [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** [Wino-X repository](https://github.com/demelin/Wino-X)
- **Repository:** [Wino-X repository](https://github.com/demelin/Wino-X)
- **Paper:** [Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution](https://aclanthology.org/2021.emnlp-main.670/)
- **Leaderboard:** [N/A]
- **Point of Contact:** [Denis Emelin](demelin.github.io)

### Dataset Summary

Wino-X is a parallel dataset of German, French, and Russian Winograd schemas, aligned with their English
counterparts, used to examine whether neural machine translation models can perform coreference resolution that
requires commonsense knowledge, and whether multilingual language models are capable of commonsense reasoning across
multiple languages.

### Supported Tasks and Leaderboards

- translation: The dataset can be used to evaluate translations of ambiguous source sentences, as produced by translation models . A [pretrained transformer-based NMT model](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) can be used for this purpose.
- coreference-resolution: The dataset can be used to rank alternative translations of an ambiguous source sentence that differ in the chosen referent of an ambiguous source pronoun. A [pretrained transformer-based NMT model](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) can be used for this purpose.
- commonsense-reasoning: The dataset can also be used evaluate whether pretrained multilingual language models can perform commonsense reasoning in (or across) multiple languages by identifying the correct filler in a cloze completion task. An [XLM-based model](https://huggingface.co/xlm-roberta-base) can be used for this purpose.

### Languages

The dataset (both its MT and LM portions) is available in the following translation pairs: English-German, English-French, English-Russian. All English sentences included in *Wino-X* were extracted from publicly available parallel corpora, as detailed in the accompanying paper, and represent the dataset-specific language varieties. All non-English sentences were obtained through machine translation and may, as such, exhibit features of translationese.

## Dataset Structure

### Data Instances

The following represents a typical *MT-Wino-X* instance (for the English-German translation pair):

{"qID": "3UDTAB6HH8D37OQL3O6F3GXEEOF09Z-1",
"sentence": "The woman looked for a different vase for the bouquet because it was too small.",
"translation1": "Die Frau suchte nach einer anderen Vase für den Blumenstrauß, weil sie zu klein war.",
"translation2": "Die Frau suchte nach einer anderen Vase für den Blumenstrauß, weil er zu klein war.",
"answer": 1,
"pronoun1": "sie",
"pronoun2": "er",
"referent1_en": "vase",
"referent2_en": "bouquet",
"true_translation_referent_of_pronoun1_de": "Vase",
"true_translation_referent_of_pronoun2_de": "Blumenstrauß",
"false_translation_referent_of_pronoun1_de": "Vase",
"false_translation_referent_of_pronoun2_de": "Blumenstrauß"}

The following represents a typical *LM-Wino-X* instance (for the English-French translation pair):

{"qID": "3UDTAB6HH8D37OQL3O6F3GXEEOF09Z-1",
"sentence": "The woman looked for a different vase for the bouquet because it was too small.",
"context_en": "The woman looked for a different vase for the bouquet because _ was too small.",
"context_fr": "La femme a cherché un vase différent pour le bouquet car _ était trop petit.",
"option1_en": "the bouquet",
"option2_en": "the vase",
"option1_fr": "le bouquet",
"option2_fr": "le vase",
"answer": 2,
"context_referent_of_option1_fr": "bouquet",
"context_referent_of_option2_fr": "vase"}

### Data Fields

For *MT-Wino-X*:

- "qID": Unique identifier ID for this dataset instance.
- "sentence": English sentence containing the ambiguous pronoun 'it'.
- "translation1": First translation candidate.
- "translation2": Second translation candidate.
- "answer": ID of the correct translation.
- "pronoun1": Translation of the ambiguous source pronoun in translation1.
- "pronoun2": Translation of the ambiguous source pronoun in translation2.
- "referent1_en": English referent of the translation of the ambiguous source pronoun in translation1.
- "referent2_en": English referent of the translation of the ambiguous source pronoun in translation2.
- "true_translation_referent_of_pronoun1_[TGT-LANG]": Target language referent of pronoun1 in the correct translation.
- "true_translation_referent_of_pronoun2_[TGT-LANG]": Target language referent of pronoun2 in the correct translation.
- "false_translation_referent_of_pronoun1_[TGT-LANG]": Target language referent of pronoun1 in the incorrect translation.
- "false_translation_referent_of_pronoun2_[TGT-LANG]": Target language referent of pronoun2 in the incorrect translation.

For *LM-Wino-X*:

- "qID": Unique identifier ID for this dataset instance.
- "sentence": English sentence containing the ambiguous pronoun 'it'.
- "context_en": Same English sentence, where 'it' is replaced by a gap.
- "context_fr": Target language translation of the English sentence, where the translation of 'it' is replaced by a gap.
- "option1_en": First filler option for the English sentence.
- "option2_en": Second filler option for the English sentence.
- "option1_[TGT-LANG]": First filler option for the target language sentence.
- "option2_[TGT-LANG]": Second filler option for the target language sentence.
- "answer": ID of the correct gap filler.
- "context_referent_of_option1_[TGT-LANG]": English translation of option1_[TGT-LANG].
- "context_referent_of_option2_[TGT-LANG]": English translation of option2_[TGT-LANG]

### Data Splits

*Wno-X* was designed as an evaluation-only benchmark and therefore is intended to be used for zero-shot testing only. However, users are very welcome to split the data as they wish :) .

## Dataset Creation

### Curation Rationale

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Source Data

#### Initial Data Collection and Normalization

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

#### Who are the source language producers?

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Annotations

#### Annotation process

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

#### Who are the annotators?

Annotations were generated automatically and verified by the dataset author / curator for correctness.

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Discussion of Biases

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Other Known Limitations

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

## Additional Information

### Dataset Curators

[Denis Emelin](demelin.github.io)

### Licensing Information

MIT

### Citation Information

@inproceedings{Emelin2021WinoXMW,
title={Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution},
author={Denis Emelin and Rico Sennrich},
booktitle={EMNLP},
year={2021}
}

原始信息汇总

数据集概述

数据集名称

• 名称: Wino-X

数据集特征

• 语言: 英语、德语、法语、俄语
• 语言创建者: 机器生成、专家生成
• 许可证: MIT
• 多语言性: 多语言、翻译
• 大小: 1K<n<10K
• 来源: 原始数据

支持的任务

• 翻译
• 指代消解
• 常识推理

数据集结构

• 数据实例: 包含MT-Wino-X和LM-Wino-X两种类型的实例，每种实例包含不同的数据字段，用于评估翻译和常识推理能力。

数据集创建

• 注释过程: 自动生成并由数据集作者/策展人验证正确性。

使用数据集的注意事项

• 许可证: MIT
• 引用信息: 请参考数据集论文的引用部分。

数据集详细信息

数据集总结

• Wino-X是一个多语言的Winograd模式数据集，用于评估神经机器翻译模型在需要常识知识的指代消解任务上的表现，以及多语言语言模型在多语言间进行常识推理的能力。

支持的任务和语言

• 任务: 翻译、指代消解、常识推理
• 语言: 英语-德语、英语-法语、英语-俄语

数据集结构

• 数据实例: 描述了MT-Wino-X和LM-Wino-X的具体数据结构和字段，用于评估翻译和常识推理任务。

数据集创建

• 注释过程: 数据集的注释是通过自动方式生成，并由数据集作者验证其正确性。

使用数据集的注意事项

• 许可证: 数据集使用MIT许可证。
• 引用信息: 提供了数据集的引用信息，方便学术引用。