Muennighoff/flores200
收藏资源简介：
--- annotations_creators: - found language_creators: - expert-generated license: - cc-by-sa-4.0 language: - ace - acm - acq - aeb - afr - ajp - aka - als - amh - apc - arb - ars - ary - arz - asm - ast - awa - ayr - azb - azj - bak - bam - ban - bel - bem - ben - bho - bjn - bod - bos - bug - bul - cat - ceb - ces - cjk - ckb - crh - cym - dan - deu - dik - dyu - dzo - ell - eng - epo - est - eus - ewe - fao - fij - fin - fon - fra - fur - fuv - gaz - gla - gle - glg - grn - guj - hat - hau - heb - hin - hne - hrv - hun - hye - ibo - ilo - ind - isl - ita - jav - jpn - kab - kac - kam - kan - kas - kat - kaz - kbp - kea - khk - khm - kik - kin - kir - kmb - kmr - knc - kon - kor - lao - lij - lim - lin - lit - lmo - ltg - ltz - lua - lug - luo - lus - lvs - mag - mai - mal - mar - min - mkd - mlt - mni - mos - mri - mya - nld - nno - nob - npi - nso - nus - nya - oci - ory - pag - pan - pap - pbt - pes - plt - pol - por - prs - quy - ron - run - rus - sag - san - sat - scn - shn - sin - slk - slv - smo - sna - snd - som - sot - spa - srd - srp - ssw - sun - swe - swh - szl - tam - taq - tat - tel - tgk - tgl - tha - tir - tpi - tsn - tso - tuk - tum - tur - twi - tzm - uig - ukr - umb - urd - uzn - vec - vie - war - wol - xho - ydd - yor - yue - zho - zsm - zul multilinguality: - multilingual - translation size_categories: - unknown source_datasets: - extended|flores task_categories: - text2text-generation - translation task_ids: [] paperswithcode_id: flores pretty_name: flores200 tags: - conditional-text-generation --- # Dataset Card for Flores200 ## Table of Contents - [Dataset Card for Flores200](#dataset-card-for-flores200) - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) ## Dataset Description - **Home:** [Flores](https://github.com/facebookresearch/flores) - **Repository:** [Github](https://github.com/facebookresearch/flores) ### Dataset Summary FLORES is a benchmark dataset for machine translation between English and low-resource languages. >The creation of FLORES200 doubles the existing language coverage of FLORES-101. Given the nature of the new languages, which have less standardization and require more specialized professional translations, the verification process became more complex. This required modifications to the translation workflow. FLORES-200 has several languages which were not translated from English. Specifically, several languages were translated from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also includes two script alternatives for four languages. FLORES-200 consists of translations from 842 distinct web articles, totaling 3001 sentences. These sentences are divided into three splits: dev, devtest, and test (hidden). On average, sentences are approximately 21 words long. **Disclaimer**: *The Flores200 dataset is hosted by the Facebook and licensed under the [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/). ### Supported Tasks and Leaderboards #### Multilingual Machine Translation Refer to the [Dynabench leaderboard](https://dynabench.org/flores/Flores%20MT%20Evaluation%20(FULL)) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on [Large-Scale Multilingual Machine Translation](http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html). Flores 200 is an extention of this. ### Languages The dataset contains parallel sentences for 200 languages, as mentioned in the original [Github](https://github.com/facebookresearch/flores/blob/master/README.md) page for the project. Languages are identified with the ISO 639-3 code (e.g. `eng`, `fra`, `rus`) plus an additional code describing the script (e.g., "eng_Latn", "ukr_Cyrl"). See [the webpage for code descriptions](https://github.com/facebookresearch/flores/blob/main/flores200/README.md). Use the configuration `all` to access the full set of parallel sentences for all the available languages in a single command. Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng_Latn-ukr_Cyrl" will provide sentences in the format below). ## Dataset Structure ### Data Instances A sample from the `dev` split for the Ukrainian language (`ukr_Cyrl` config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits. ```python { 'id': 1, 'sentence': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.', 'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 'domain': 'wikinews', 'topic': 'health', 'has_image': 0, 'has_hyperlink': 0 } ``` When using a hyphenated pairing or using the `all` function, data will be presented as follows: ```python { 'id': 1, 'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 'domain': 'wikinews', 'topic': 'health', 'has_image': 0, 'has_hyperlink': 0, 'sentence_eng_Latn': 'On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.', 'sentence_ukr_Cyrl': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.' } ``` The text is provided as-in the original dataset, without further preprocessing or tokenization. ### Data Fields - `id`: Row number for the data entry, starting at 1. - `sentence`: The full sentence in the specific language (may have _lang for pairings) - `URL`: The URL for the English article from which the sentence was extracted. - `domain`: The domain of the sentence. - `topic`: The topic of the sentence. - `has_image`: Whether the original article contains an image. - `has_hyperlink`: Whether the sentence contains a hyperlink. ### Data Splits | config| `dev`| `devtest`| |-----------------:|-----:|---------:| |all configurations| 997| 1012:| ### Dataset Creation Please refer to the original article [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) for additional information on dataset creation. ## Additional Information ### Dataset Curators See paper for details. ### Licensing Information Licensed with Creative Commons Attribution Share Alike 4.0. License available [here](https://creativecommons.org/licenses/by-sa/4.0/). ### Citation Information Please cite the authors if you use these corpora in your work: ```bibtex @article{nllb2022, author = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang}, title = {No Language Left Behind: Scaling Human-Centered Machine Translation}, year = {2022} } ```
annotations_creators: - 公开采集（found） language_creators: - 专家人工生成 license: - CC BY-SA 4.0 language: - ace - acm - acq - aeb - afr - ajp - aka - als - amh - apc - arb - ars - ary - arz - asm - ast - awa - ayr - azb - azj - bak - bam - ban - bel - bem - ben - bho - bjn - bod - bos - bug - bul - cat - ceb - ces - cjk - ckb - crh - cym - dan - deu - dik - dyu - dzo - ell - eng - epo - est - eus - ewe - fao - fij - fin - fon - fra - fur - fuv - gaz - gla - gle - glg - grn - guj - hat - hau - heb - hin - hne - hrv - hun - hye - ibo - ilo - ind - isl - ita - jav - jpn - kab - kac - kam - kan - kas - kat - kaz - kbp - kea - khk - khm - kik - kin - kir - kmb - kmr - knc - kon - kor - lao - lij - lim - lin - lit - lmo - ltg - ltz - lua - lug - luo - lus - lvs - mag - mai - mal - mar - min - mkd - mlt - mni - mos - mri - mya - nld - nno - nob - npi - nso - nus - nya - oci - ory - pag - pan - pap - pbt - pes - plt - pol - por - prs - quy - ron - run - rus - sag - san - sat - scn - shn - sin - slk - slv - smo - sna - snd - som - sot - spa - srd - srp - ssw - sun - swe - swh - szl - tam - taq - tat - tel - tgk - tgl - tha - tir - tpi - tsn - tso - tuk - tum - tur - twi - tzm - uig - ukr - umb - urd - uzn - vec - vie - war - wol - xho - ydd - yor - yue - zho - zsm - zul multilinguality: - 多语言 - 翻译任务 size_categories: - 未知 source_datasets: - 扩展|flores task_categories: - 文本到文本生成（text2text-generation） - 机器翻译 task_ids: [] paperswithcode_id: flores pretty_name: flores200 tags: - 条件文本生成（conditional-text-generation） # Flores200 数据集卡片 ## 目录 - [Flores200 数据集卡片](#dataset-card-for-flores200) - [目录](#table-of-contents) - [数据集描述](#dataset-description) - [数据集概览](#dataset-summary) - [支持任务与评测榜单](#supported-tasks-and-leaderboards) - [语言覆盖](#languages) - [数据集结构](#dataset-structure) - [数据样例](#data-instances) - [数据字段](#data-fields) - [数据划分](#data-splits) - [数据集构建](#dataset-creation) - [附加信息](#additional-information) - [数据集维护者](#dataset-curators) - [许可信息](#licensing-information) - [引用信息](#citation-information) ## 数据集描述 - **主页**：[Flores](https://github.com/facebookresearch/flores) - **代码仓库**：[GitHub](https://github.com/facebookresearch/flores) ### 数据集概览 FLORES是用于英语与低资源语言间机器翻译的基准数据集。 > FLORES200的语言覆盖量是FLORES-101的两倍。由于新增语言的标准化程度较低，需要更专业的人工翻译，因此验证流程更为复杂，这也要求对翻译工作流进行调整。FLORES-200包含多种并非从英语翻译而来的语言：具体而言，部分语言的翻译源为西班牙语、法语、俄语与现代标准阿拉伯语。此外，FLORES-200还为四种语言提供了两种书写脚本变体。FLORES-200的语料源自842篇不同的网络文章，总计3001个句子，被划分为dev、devtest与test（隐藏）三个子集。句子的平均长度约为21个单词。 **免责声明**：*Flores200数据集由Facebook托管，采用[知识共享署名-相同方式共享4.0国际许可协议](https://creativecommons.org/licenses/by-sa/4.0/)进行授权。* ### 支持任务与评测榜单 #### 多语言机器翻译 有关FLORES-101在WMT2021「大规模多语言机器翻译」共享任务中的模型评测细节，请参考[Dynabench评测榜单](https://dynabench.org/flores/Flores%20MT%20Evaluation%20(FULL))。Flores 200是该任务的扩展数据集。 ### 语言覆盖 本数据集覆盖200种语言的平行语料，详情可参考项目原始[GitHub页面](https://github.com/facebookresearch/flores/blob/master/README.md)。语言采用ISO 639-3代码标识（例如`eng`、`fra`、`rus`），并附加脚本描述代码（例如`eng_Latn`、`ukr_Cyrl`）。具体代码含义可查阅[代码说明页面](https://github.com/facebookresearch/flores/blob/main/flores200/README.md)。 可通过配置`all`一次性获取所有可用语言的完整平行语料； 可通过连字符连接的语言对获取指定双语语料（例如`eng_Latn-ukr_Cyrl`将返回如下格式的语料）。 ## 数据集结构 ### 数据样例 以下展示了乌克兰语（`ukr_Cyrl`配置）`dev`子集的一条样例。所有配置的结构均保持一致，所有句子在不同配置与子集间均保持对齐。 python { 'id': 1, 'sentence': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.', 'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 'domain': 'wikinews', 'topic': 'health', 'has_image': 0, 'has_hyperlink': 0 } 当使用语言对配置或`all`配置时，数据将以如下格式呈现： python { 'id': 1, 'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 'domain': 'wikinews', 'topic': 'health', 'has_image': 0, 'has_hyperlink': 0, 'sentence_eng_Latn': 'On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.', 'sentence_ukr_Cyrl': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.' } 文本内容直接保留自原始数据集，未进行额外的预处理或分词操作。 ### 数据字段 - `id`：数据条目的行号，从1开始计数。 - `sentence`：指定语言的完整句子（若为语言对配置则会带有`_lang`后缀）。 - `URL`：提取该句子的英文原文文章的链接。 - `domain`：句子所属的领域。 - `topic`：句子所属的主题。 - `has_image`：原始文章是否包含图片。 - `has_hyperlink`：该句子是否包含超链接。 ### 数据划分 | 配置| `dev`| `devtest`| |-----------------:|-----:|---------:| |所有配置| 997| 1012| ### 数据集构建 如需了解数据集构建的更多细节，请查阅原论文[《No Language Left Behind: Scaling Human-Centered Machine Translation》](https://arxiv.org/abs/2207.04672)。 ## 附加信息 ### 数据集维护者 详情请查阅相关论文。 ### 许可信息 本数据集采用知识共享署名-相同方式共享4.0协议进行授权，许可协议详情可参见[此处](https://creativecommons.org/licenses/by-sa/4.0/)。 ### 引用信息 若在研究中使用该语料，请引用如下文献： bibtex @article{nllb2022, author = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang}, title = {No Language Left Behind: Scaling Human-Centered Machine Translation}, year = {2022} }
数据集概述
数据集名称
- 名称: Flores200
数据集描述
- 描述: Flores200是一个用于机器翻译的基准数据集，专注于英语与低资源语言之间的翻译。该数据集扩展了原有的FLORES-101，增加了更多的语言覆盖和翻译复杂性。
语言支持
- 语言数量: 200种语言
- 语言示例: ace, acm, acq, aeb, afr, ajp, aka, als, amh, apc, arb, ars, ary, arz, asm, ast, awa, ayr, azb, azj, bak, bam, ban, bel, bem, ben, bho, bjn, bod, bos, bug, bul, cat, ceb, ces, cjk, ckb, crh, cym, dan, deu, dik, dyu, dzo, ell, eng, epo, est, eus, ewe, fao, fij, fin, fon, fra, fur, fuv, gaz, gla, gle, glg, grn, guj, hat, hau, heb, hin, hne, hrv, hun, hye, ibo, ilo, ind, isl, ita, jav, jpn, kab, kac, kam, kan, kas, kat, kaz, kbp, kea, khk, khm, kik, kin, kir, kmb, kmr, knc, kon, kor, lao, lij, lim, lin, lit, lmo, ltg, ltz, lua, lug, luo, lus, lvs, mag, mai, mal, mar, min, mkd, mlt, mni, mos, mri, mya, nld, nno, nob, npi, nso, nus, nya, oci, ory, pag, pan, pap, pbt, pes, plt, pol, por, prs, quy, ron, run, rus, sag, san, sat, scn, shn, sin, slk, slv, smo, sna, snd, som, sot, spa, srd, srp, ssw, sun, swe, swh, szl, tam, taq, tat, tel, tgk, tgl, tha, tir, tpi, tsn, tso, tuk, tum, tur, twi, tzm, uig, ukr, umb, urd, uzn, vec, vie, war, wol, xho, ydd, yor, yue, zho, zsm, zul
数据集结构
- 数据实例: 每个实例包含id, sentence, URL, domain, topic, has_image, has_hyperlink等字段。
- 数据字段:
id: 数据条目的行号，从1开始。sentence: 特定语言的完整句子。URL: 英文文章的URL。domain: 句子的领域。topic: 句子的主题。has_image: 原文是否包含图片。has_hyperlink: 句子是否包含超链接。
数据集创建
- 创建过程: 数据集的创建涉及从842个不同的网络文章中提取的3001个句子，分为dev, devtest, 和test（隐藏）三个部分。平均每个句子约21个单词。
许可证信息
- 许可证: Creative Commons Attribution-ShareAlike 4.0 International License
引用信息
- 引用格式: bibtex @article{nllb2022, author = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang}, title = {No Language Left Behind: Scaling Human-Centered Machine Translation}, year = {2022} }
