neulab/ted_multi
收藏资源简介：
TEDMulti数据集是一个包含60种语言的大规模多语言数据集，源自TED演讲的转录文本。每个记录由语言和文本的并行数组组成，缺失和不完整的翻译会被过滤掉。数据集包含训练集、验证集和测试集，分别包含258098、6049和7213个样本。数据集的下载文件大小为352.23 MB，生成的数据集大小为791.01 MB，总磁盘使用量为1.14 GB。数据集的字段包括`translations`（多语言字符串）和`talk_name`（字符串）。
TEDMulti数据集是一个包含60种语言的大规模多语言数据集，源自TED演讲的转录文本。每个记录由语言和文本的并行数组组成，缺失和不完整的翻译会被过滤掉。数据集包含训练集、验证集和测试集，分别包含258098、6049和7213个样本。数据集的下载文件大小为352.23 MB，生成的数据集大小为791.01 MB，总磁盘使用量为1.14 GB。数据集的字段包括`translations`（多语言字符串）和`talk_name`（字符串）。
数据集概述
数据集名称
- pretty_name: TEDMulti
数据集特征
- features:
- name: translations
- dtype: multilingual string
- languages: 60 languages including
ar,az,be,bg,bn, etc.
- name: talk_name
- dtype: string
- name: translations
数据集配置
- config_name: plain_text
数据集分割
- splits:
- name: test
- num_bytes: 23364983
- num_examples: 7213
- name: train
- num_bytes: 748209995
- num_examples: 258098
- name: validation
- num_bytes: 19435383
- num_examples: 6049
- name: test
数据集大小
- download_size: 352222045
- dataset_size: 791010361
数据集结构
数据实例
-
示例:
{ "talk_name": "shabana_basij_rasikh_dare_to_educate_afghan_girls", "translations": "{"language": ["ar", "az", "bg", "bn", "cs", "da", "de", "el", "en", "es", "fa", "fr", "he", "hi", "hr", "hu", "hy", "id", "it", ..." }
数据字段
- plain_text:
- translations: multilingual string
- talk_name: string
数据分割
-
splits:
name train validation test plain_text 258098 6049 7213
数据集创建
数据集来源
- source: TED Talk transcripts
数据集描述
- summary: Massively multilingual (60 language) data set derived from TED Talk transcripts. Each record consists of parallel arrays of language and text. Missing and incomplete translations will be filtered out.
引用信息
@InProceedings{qi-EtAl:2018:N18-2, author = {Qi, Ye and Sachan, Devendra and Felix, Matthieu and Padmanabhan, Sarguna and Neubig, Graham}, title = {When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?}, booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)}, month = {June}, year = {2018}, address = {New Orleans, Louisiana}, publisher = {Association for Computational Linguistics}, pages = {529--535}, abstract = {The performance of Neural Machine Translation (NMT) systems often suffers in low-resource scenarios where sufficiently large-scale parallel corpora cannot be obtained. Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks, which often suffer from paucity of data. However, their utility for NMT has not been extensively explored. In this work, we perform five sets of experiments that analyze when we can expect pre-trained word embeddings to help in NMT tasks. We show that such embeddings can be surprisingly effective in some cases -- providing gains of up to 20 BLEU points in the most favorable setting.}, url = {http://www.aclweb.org/anthology/N18-2084} }
