opus_dgt

# Dataset Card for OPUS DGT

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://opus.nlpl.eu/DGT/corpus/version/DGT
- **Homepage:** https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** https://aclanthology.org/L12-1481/
- **Paper:** https://aclanthology.org/L12-1246/
- **Leaderboard:** [More Information Needed]
- **Point of Contact:** [More Information Needed]

### Dataset Summary

A collection of translation memories provided by the Joint Research Centre (JRC) Directorate-General for Translation (DGT): https://ec.europa.eu/jrc/en/language-technologies/dgt-translation-memory

Latest Release: v2019.

Tha dataset contains 25 languages and 299 bitexts.

To load a language pair which isn't part of the config, all you need to do is specify the language code as pairs,
e.g.

```python
dataset = load_dataset("opus_dgt", lang1="it", lang2="pl")
```

You can find the valid pairs in Homepage section of Dataset Description: https://opus.nlpl.eu/DGT/corpus/version/DGT

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

The languages in the dataset are:
- bg
- cs
- da
- de
- el
- en
- es
- et
- fi
- fr
- ga
- hr
- hu
- it
- lt
- lv
- mt
- nl
- pl
- pt
- ro
- sh
- sk
- sl
- sv

## Dataset Structure

### Data Instances

```
{
'id': '0',
'translation': {
"bg": "Протокол за поправка на Конвенцията относно компетентността, признаването и изпълнението на съдебни решения по граждански и търговски дела, подписана в Лугано на 30 октомври 2007 г.",
"ga": "Miontuairisc cheartaitheach maidir le Coinbhinsiún ar dhlínse agus ar aithint agus ar fhorghníomhú breithiúnas in ábhair shibhialta agus tráchtála, a siníodh in Lugano an 30 Deireadh Fómhair 2007"
}
}
```

### Data Fields

- `id` (`str`): Unique identifier of the parallel sentence for the pair of languages.
- `translation` (`dict`): Parallel sentences for the pair of languages.

### Data Splits

The dataset contains a single `train` split.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

[More Information Needed]

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

[More Information Needed]

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

The conditions of use of the original source dataset are: https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en#conditions
> The DGT-TM database is the exclusive property of the European Commission. The Commission cedes its non-exclusive rights free of charge and world-wide for the entire duration of the protection of those rights to the re-user, for all kinds of use which comply with the conditions laid down in the Commission Decision of 12 December 2011 on the re-use of Commission documents, published in Official Journal of the European Union L330 of 14 December 2011, pages 39 to 42.

> Any re-use of the database or of the structured elements contained in it is required to be identified by the re-user, who is under an obligation to state the source of the documents used: the website address, the date of the latest update and the fact that the European Commission retains ownership of the data.

### Citation Information

https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en#reference

When making reference to DGT-TM in scientific publications, please refer to:
```bibtex
@inproceedings{steinberger-etal-2012-dgt,
title = "{DGT}-{TM}: A freely available Translation Memory in 22 languages",
author = {Steinberger, Ralf and
Eisele, Andreas and
Klocek, Szymon and
Pilos, Spyridon and
Schl{\"u}ter, Patrick},
editor = "Calzolari, Nicoletta and
Choukri, Khalid and
Declerck, Thierry and
Do{\u{g}}an, Mehmet U{\u{g}}ur and
Maegaard, Bente and
Mariani, Joseph and
Moreno, Asuncion and
Odijk, Jan and
Piperidis, Stelios",
booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
month = may,
year = "2012",
address = "Istanbul, Turkey",
publisher = "European Language Resources Association (ELRA)",
url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/814_Paper.pdf",
pages = "454--459",
}

```

Please cite the following article if you use any part of the corpus in your own work:
```bibtex
@inproceedings{tiedemann-2012-parallel,
title = "Parallel Data, Tools and Interfaces in {OPUS}",
author = {Tiedemann, J{\"o}rg},
editor = "Calzolari, Nicoletta and
Choukri, Khalid and
Declerck, Thierry and
Do{\u{g}}an, Mehmet U{\u{g}}ur and
Maegaard, Bente and
Mariani, Joseph and
Moreno, Asuncion and
Odijk, Jan and
Piperidis, Stelios",
booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
month = may,
year = "2012",
address = "Istanbul, Turkey",
publisher = "European Language Resources Association (ELRA)",
url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf",
pages = "2214--2218",
}
```

### Contributions

Thanks to [@rkc007](https://github.com/rkc007) for adding this dataset.

# OPUS DGT 数据集卡片（Dataset Card）

## 目录（Table of Contents）
- [数据集描述（Dataset Description）](#dataset-description)
- [数据集摘要（Dataset Summary）](#dataset-summary)
- [支持任务与排行榜（Supported Tasks and Leaderboards）](#supported-tasks-and-leaderboards)
- [语言（Languages）](#languages)
- [数据集结构（Dataset Structure）](#dataset-structure)
- [数据实例（Data Instances）](#data-instances)
- [数据字段（Data Fields）](#data-fields)
- [数据划分（Data Splits）](#data-splits)
- [数据集构建（Dataset Creation）](#dataset-creation)
- [数据集遴选依据（Curation Rationale）](#curation-rationale)
- [源数据（Source Data）](#source-data)
- [标注（Annotations）](#annotations)
- [个人与敏感信息（Personal and Sensitive Information）](#personal-and-sensitive-information)
- [数据使用注意事项（Considerations for Using the Data）](#considerations-for-using-the-data)
- [数据集的社会影响（Social Impact of Dataset）](#social-impact-of-dataset)
- [偏差讨论（Discussion of Biases）](#discussion-of-biases)
- [其他已知局限（Other Known Limitations）](#other-known-limitations)
- [附加信息（Additional Information）](#additional-information)
- [数据集遴选者（Dataset Curators）](#dataset-curators)
- [许可信息（Licensing Information）](#licensing-information)
- [引用信息（Citation Information）](#citation-information)
- [贡献（Contributions）](#contributions)

## 数据集描述（Dataset Description）

- **主页（Homepage）:** https://opus.nlpl.eu/DGT/corpus/version/DGT
- **主页（Homepage）:** https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en
- **代码仓库（Repository）:** [需补充更多信息](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **论文（Paper）:** https://aclanthology.org/L12-1481/
- **论文（Paper）:** https://aclanthology.org/L12-1246/
- **排行榜（Leaderboard）:** [需补充更多信息]
- **联络人（Point of Contact）:** [需补充更多信息]

### 数据集摘要（Dataset Summary）

本数据集为欧洲委员会联合研究中心（Joint Research Centre, JRC）翻译总司（Directorate-General for Translation, DGT）提供的翻译记忆库（Translation Memory）集合，详情见：https://ec.europa.eu/jrc/en/language-technologies/dgt-translation-memory

最新版本：v2019。

本数据集涵盖25种语言，包含299组双语文本对。

若需加载配置中未包含的语言对，仅需按如下格式指定语言代码，示例如下：
python
dataset = load_dataset("opus_dgt", lang1="it", lang2="pl")

可在数据集描述的主页部分查看有效语言对列表：https://opus.nlpl.eu/DGT/corpus/version/DGT

### 支持任务与排行榜（Supported Tasks and Leaderboards）

[需补充更多信息]

### 语言（Languages）

本数据集包含的语言如下：
- bg
- cs
- da
- de
- el
- en
- es
- et
- fi
- fr
- ga
- hr
- hu
- it
- lt
- lv
- mt
- nl
- pl
- pt
- ro
- sh
- sk
- sl
- sv

## 数据集结构（Dataset Structure）

### 数据实例（Data Instances）

{
'id': '0',
'translation': {
"bg": "Протокол за поправка на Конвенцията относно компетентността, признаването и изпълнението на съдебни решения по граждански и търговски дела, подписана в Лугано на 30 октомври 2007 г.",
"ga": "Miontuairisc cheartaitheach maidir le Coinbhinsiún ar dhlínse agus ar aithint agus ar fhorghníomhú breithiúnas in ábhair shibhialta agus tráchtála, a siníodh in Lugano an 30 Deireadh Fómhair 2007"
}
}

### 数据字段（Data Fields）

- `id` (`str`): 该语言对平行句对的唯一标识符。
- `translation` (`dict`): 对应语言对的平行语句集合。

### 数据划分（Data Splits）

本数据集仅包含一个`train`划分。

## 数据集构建（Dataset Creation）

### 数据集遴选依据（Curation Rationale）

[需补充更多信息]

### 源数据（Source Data）

[需补充更多信息]

#### 初始数据收集与标准化（Initial Data Collection and Normalization）

[需补充更多信息]

#### 源语言生成者是谁？（Who are the source language producers?）

[需补充更多信息]

### 标注（Annotations）

[需补充更多信息]

#### 标注流程（Annotation process）

[需补充更多信息]

#### 标注者是谁？（Who are the annotators?）

[需补充更多信息]

### 个人与敏感信息（Personal and Sensitive Information）

[需补充更多信息]

## 数据使用注意事项（Considerations for Using the Data）

### 数据集的社会影响（Social Impact of Dataset）

[需补充更多信息]

### 偏差讨论（Discussion of Biases）

[需补充更多信息]

### 其他已知局限（Other Known Limitations）

[需补充更多信息]

## 附加信息（Additional Information）

### 数据集遴选者（Dataset Curators）

[需补充更多信息]

### 许可信息（Licensing Information）

原始源数据集的使用条款见：https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en#conditions
> DGT-TM数据库为欧洲委员会专属财产。欧洲委员会将其非独家使用权在权利保护期内全球范围内免费授予再使用者，所有使用行为需符合2011年12月12日欧盟委员会关于委员会文档再利用的决定中规定的条款，该决定刊载于2011年12月14日《欧盟官方公报》L330卷第39至42页。

> 对该数据库或其中包含的结构化元素进行任何再利用时，再使用者必须标注来源，并需明确说明所使用文档的来源：网站地址、最新更新日期，以及欧洲委员会保留该数据所有权这一事实。

### 引用信息（Citation Information）

引用参考链接：https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en#reference

若在学术出版物中引用DGT-TM，请参考以下文献：
bibtex
@inproceedings{steinberger-etal-2012-dgt,
title = "{DGT}-{TM}: A freely available Translation Memory in 22 languages",
author = {Steinberger, Ralf and
Eisele, Andreas and
Klocek, Szymon and
Pilos, Spyridon and
Schl{"u}ter, Patrick},
editor = "Calzolari, Nicoletta and
Choukri, Khalid and
Declerck, Thierry and
Do{u{g}}an, Mehmet U{u{g}}ur and
Maegaard, Bente and
Mariani, Joseph and
Moreno, Asuncion and
Odijk, Jan and
Piperidis, Stelios",
booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
month = may,
year = "2012",
address = "Istanbul, Turkey",
publisher = "European Language Resources Association (ELRA)",
url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/814_Paper.pdf",
pages = "454--459",
}

若您的研究中使用了该语料库的任意部分，请引用以下文献：
bibtex
@inproceedings{tiedemann-2012-parallel,
title = "Parallel Data, Tools and Interfaces in {OPUS}",
author = {Tiedemann, J{"o}rg},
editor = "Calzolari, Nicoletta and
Choukri, Khalid and
Declerck, Thierry and
Do{u{g}}an, Mehmet U{u{g}}ur and
Maegaard, Bente and
Mariani, Joseph and
Moreno, Asuncion and
Odijk, Jan and
Piperidis, Stelios",
booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
month = may,
year = "2012",
address = "Istanbul, Turkey",
publisher = "European Language Resources Association (ELRA)",
url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf",
pages = "2214--2218",
}

### 贡献（Contributions）

感谢 [@rkc007](https://github.com/rkc007) 为本数据集添加的贡献。