wmt/wmt15

---
annotations_creators:
- no-annotation
language_creators:
- found
language:
- cs
- de
- en
- fi
- fr
- ru
license:
- unknown
multilinguality:
- translation
size_categories:
- 10M<n<100M
source_datasets:
- extended|europarl_bilingual
- extended|giga_fren
- extended|news_commentary
- extended|un_multi
task_categories:
- translation
task_ids: []
paperswithcode_id: wmt-2015
pretty_name: WMT15
dataset_info:
- config_name: cs-en
features:
- name: translation
dtype:
translation:
languages:
- cs
- en
splits:
- name: train
num_bytes: 282996174
num_examples: 959768
- name: validation
num_bytes: 757809
num_examples: 3003
- name: test
num_bytes: 572195
num_examples: 2656
download_size: 170092459
dataset_size: 284326178
- config_name: de-en
features:
- name: translation
dtype:
translation:
languages:
- de
- en
splits:
- name: train
num_bytes: 1363999245
num_examples: 4522998
- name: validation
num_bytes: 777326
num_examples: 3003
- name: test
num_bytes: 522981
num_examples: 2169
download_size: 821722473
dataset_size: 1365299552
- config_name: fi-en
features:
- name: translation
dtype:
translation:
languages:
- fi
- en
splits:
- name: train
num_bytes: 605145153
num_examples: 2073394
- name: validation
num_bytes: 363933
num_examples: 1500
- name: test
num_bytes: 306327
num_examples: 1370
download_size: 347635272
dataset_size: 605815413
- config_name: fr-en
features:
- name: translation
dtype:
translation:
languages:
- fr
- en
splits:
- name: train
num_bytes: 14758953934
num_examples: 40853137
- name: validation
num_bytes: 1138729
num_examples: 4503
- name: test
num_bytes: 298763
num_examples: 1500
download_size: 7781794388
dataset_size: 14760391426
- config_name: ru-en
features:
- name: translation
dtype:
translation:
languages:
- ru
- en
splits:
- name: train
num_bytes: 437751056
num_examples: 1495081
- name: validation
num_bytes: 1087738
num_examples: 3003
- name: test
num_bytes: 955964
num_examples: 2818
download_size: 225905330
dataset_size: 439794758
configs:
- config_name: cs-en
data_files:
- split: train
path: cs-en/train-*
- split: validation
path: cs-en/validation-*
- split: test
path: cs-en/test-*
- config_name: de-en
data_files:
- split: train
path: de-en/train-*
- split: validation
path: de-en/validation-*
- split: test
path: de-en/test-*
- config_name: fi-en
data_files:
- split: train
path: fi-en/train-*
- split: validation
path: fi-en/validation-*
- split: test
path: fi-en/test-*
- config_name: fr-en
data_files:
- split: train
path: fr-en/train-*
- split: validation
path: fr-en/validation-*
- split: test
path: fr-en/test-*
- config_name: ru-en
data_files:
- split: train
path: ru-en/train-*
- split: validation
path: ru-en/validation-*
- split: test
path: ru-en/test-*
---

# Dataset Card for "wmt15"

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [http://www.statmt.org/wmt15/translation-task.html](http://www.statmt.org/wmt15/translation-task.html)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 1.74 GB
- **Size of the generated dataset:** 284.34 MB
- **Total amount of disk used:** 2.02 GB

### Dataset Summary

<div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400">
<p><b>Warning:</b> There are issues with the Common Crawl corpus data (<a href="https://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz">training-parallel-commoncrawl.tgz</a>):</p>
<ul>
<li>Non-English files contain many English sentences.</li>
<li>Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.</li>
</ul>
<p>We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains to the expectation that such data has been superseded, primarily by CCMatrix, and to some extent, by ParaCrawl datasets.</p>
</div>

Translation dataset based on the data from statmt.org.

Versions exist for different years using a combination of data
sources. The base `wmt` allows you to create a custom dataset by choosing
your own data/language pair. This can be done as follows:

```python
from datasets import inspect_dataset, load_dataset_builder

inspect_dataset("wmt15", "path/to/scripts")
builder = load_dataset_builder(
"path/to/scripts/wmt_utils.py",
language_pair=("fr", "de"),
subsets={
datasets.Split.TRAIN: ["commoncrawl_frde"],
datasets.Split.VALIDATION: ["euelections_dev2019"],
},
)

# Standard version
builder.download_and_prepare()
ds = builder.as_dataset()

# Streamable version
ds = builder.as_streaming_dataset()
```

### Supported Tasks and Leaderboards

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Languages

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Dataset Structure

### Data Instances

#### cs-en

- **Size of downloaded dataset files:** 1.74 GB
- **Size of the generated dataset:** 284.34 MB
- **Total amount of disk used:** 2.02 GB

An example of 'validation' looks as follows.
```

```

### Data Fields

The data fields are the same among all splits.

#### cs-en
- `translation`: a multilingual `string` variable, with possible languages including `cs`, `en`.

### Data Splits

|name |train |validation|test|
|-----|-----:|---------:|---:|
|cs-en|959768| 3003|2656|

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Citation Information

```

@InProceedings{bojar-EtAl:2015:WMT,
author = {Bojar, Ond
{r}ej and Chatterjee, Rajen and Federmann, Christian and Haddow, Barry and Huck, Matthias and Hokamp, Chris and Koehn, Philipp and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Post, Matt and Scarton, Carolina and Specia, Lucia and Turchi, Marco},
title = {Findings of the 2015 Workshop on Statistical Machine Translation},
booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},
month = {September},
year = {2015},
address = {Lisbon, Portugal},
publisher = {Association for Computational Linguistics},
pages = {1--46},
url = {http://aclweb.org/anthology/W15-3001}
}

```

### Contributions

Thanks to [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten) for adding this dataset.

原始信息汇总

数据集概述

数据集名称: WMT15 支持语言: cs, de, en, fi, fr, ru 许可证: 未知 多语言性: 翻译 大小类别: 10M<n<100M 任务类别: 翻译 数据来源: 扩展自europarl_bilingual, giga_fren, news_commentary, un_multi

数据集结构

配置: cs-en

• 特征:
  • 翻译: 语言包括cs, en
• 分割:
  • 训练: 959768个例子, 282996174字节
  • 验证: 3003个例子, 757809字节
  • 测试: 2656个例子, 572195字节
• 下载大小: 170092459字节
• 数据集大小: 284326178字节

配置: de-en

• 特征:
  • 翻译: 语言包括de, en
• 分割:
  • 训练: 4522998个例子, 1363999245字节
  • 验证: 3003个例子, 777326字节
  • 测试: 2169个例子, 522981字节
• 下载大小: 821722473字节
• 数据集大小: 1365299552字节

配置: fi-en

• 特征:
  • 翻译: 语言包括fi, en
• 分割:
  • 训练: 2073394个例子, 605145153字节
  • 验证: 1500个例子, 363933字节
  • 测试: 1370个例子, 306327字节
• 下载大小: 347635272字节
• 数据集大小: 605815413字节

配置: fr-en

• 特征:
  • 翻译: 语言包括fr, en
• 分割:
  • 训练: 40853137个例子, 14758953934字节
  • 验证: 4503个例子, 1138729字节
  • 测试: 1500个例子, 298763字节
• 下载大小: 7781794388字节
• 数据集大小: 14760391426字节

配置: ru-en

• 特征:
  • 翻译: 语言包括ru, en
• 分割:
  • 训练: 1495081个例子, 437751056字节
  • 验证: 3003个例子, 1087738字节
  • 测试: 2818个例子, 955964字节
• 下载大小: 225905330字节
• 数据集大小: 439794758字节

搜集汇总

背景与挑战

背景概述

wmt/wmt15是一个多语言机器翻译数据集，包含捷克语、德语、芬兰语、法语和俄语与英语的平行文本对，规模在10M到100M之间。数据集存在部分数据质量问题，如非英语文件中包含大量英语句子和平行句子未对齐的情况。