遇见数据集

BSC-LT/MULTI_corpus

收藏
Hugging Face2026-05-21 更新2026-06-14 收录
官方服务:

资源简介:

--- language: - zgh - ps - wo - rom - en - es - fr multilinguality: - multilingual pretty_name: "MULTI-Corpus" size_categories: - 10M<n<100M task_categories: - translation license: cc-by-sa-4.0 --- # Dataset Card for MULTI-Corpus ## Dataset Description - **Point of Contact:** ai_institute_mt@bsc.es ### Dataset Summary This corpus was compiled as part of the TRAIN project (*Traducción Automática para la Inclusión*, Automatic Translation for Inclusion), funded by MCIN/AEI and ERDF. It aggregates **15,191,441 sentence-level entries** covering four extremely low-resource languages: Tamazight/Amazigh (ZGH), Pashto (PS), Wolof (WO), and Romani (ROM), paired with one or more high-resource counterparts (English, Spanish, French), plus monolingual data for each low-resource language. The dataset is intended primarily for training machine translation (MT) systems targeting under-resourced languages of migrant and minority communities in Spain. It consolidates 30 source datasets drawn from OPUS, the NLLB collection, CommonVoice, Tatoeba, Col·lectivaT, IRCAM, and other sources into a single unified large dataset (see Source Data section for the full list). The data spans a wide range of domains including web-crawled text, software localisation, news, encyclopaedic content (Wikipedia), religious text (Bible), educational contents, and institutional documents. ### Supported Tasks and Leaderboards - **Machine Translation:** parallel sentence pairs for the language pairs EN-ZGH, ES-ZGH, FR-ZGH, EN-PS, ES-PS, EN-WO, FR-WO, EN-ROM, ES-ROM. ### Languages | Language | ISO code | Script | Notes | |---|---|---|---| | Tamazight (Standard Moroccan Amazigh) | `zgh` | Tifinagh (Unicode U+2D30-U+2D7F) | Recently standardised; inconsistent orthographic conventions exist across dialects | | Pashto | `ps` | Pashto/Perso-Arabic (right-to-left) | Variant of the Arabic alphabet adapted for Pashto-specific sounds | | Wolof | `wo` | Latin | Standard Latin alphabet adapted during the colonial period | | Romani | `rom` | Latin with diacritics | Diacritical marks added to represent sounds absent from standard Latin languages; significant dialectal variation | | English | `en` | Latin | High-resource parallel counterpart | | Spanish | `es` | Latin | High-resource parallel counterpart | | French | `fr` | Latin | High-resource parallel counterpart | ## Dataset Structure This dataset is provided in two different versions: The `multi-corpus-raw.tsv` file is the unfiltered aggregation of all 30 source datasets listed in the Source Data section. It contains only **original (non-synthetic)** data, covering original parallel pairs (14,493,051 rows) and monolingual sentences (698,390 rows) across the four low-resource languages ZGH, PS, WO, and ROM, paired with EN, ES, and FR. No filtering, deduplication, or synthetic data generation has been applied; the file preserves the data exactly as found in the source corpora. It is suited for exploratory analysis, custom filtering pipelines, and as the upstream source for the training set. The `multi-training-set.json` file is a preprocessed, filtered, and augmented training-ready dataset derived from the raw corpus. It contains Spanish (ES) sentences paired with sentences in ZGH, PS, WO, and ROM, and was produced through three steps: (1) filtering the raw corpus with the language identification tool [Fasttext](https://github.com/facebookresearch/fastText), which significantly reduced the size of the noisier web-crawled datasets; (2) generating synthetic ES-XX parallel data by automatically translating monolingual low-resource sentences into Spanish using [Google Translate](https://translate.google.com/); and (3) augmenting the ES-XX parallel data by pivot-translating existing EN-XX and FR-XX sentence pairs into Spanish. ### Data Instances The dataset `multi-corpus-raw.tsv` is provided in TSV (tab-separated values) format. It contains one sentence per row with the following structure: ```json { "type": "Parallel", "lang_low": "ZGH", "lang_high": "EN", "dataset": "commonvoice", "text_low": "ⴰⵔⴳⴰⵜ ⵙ ⵓⴳⵕⴹ ⵏⵏⵓⵏ", "text_high": "Contribute Your Voice" } ``` Example monolingual row: ```json { "type": "Mono", "lang_low": "ROM", "lang_high": "-", "dataset": "curriculum-framework-for-romani", "text_low": "I Romani čhib si po drom te xasavol, te na avel la protekcija thaj ažutipe lovenca katar le nacionalni thaj internacionalni aktorura.", "text_high": "-" } ``` ### Data Fields - `type`: `Parallel` (bilingual sentence pair) or `Mono` (monolingual sentence). - `lang_low`: ISO code of the low-resource language (`ZGH`, `PS`, `WO`, `ROM`). - `lang_high`: ISO code of the high-resource paired language (`EN`, `ES`, `FR`), or `-` for monolingual entries. - `dataset`: Name of the source dataset (see Source Data section for the full list). - `text_low`: Sentence in the low-resource language. - `text_high`: Paired sentence in the high-resource language, or `-` for monolingual entries. The file `multi-training-set.json` is provided in JSON format. The structure follows the input format required by [MADLAD-400](https://huggingface.co/google/madlad400-3b-mt), the multilingual MT model used for experimentation in the TRAIN project. Specifically, the target low-resource language is indicated by a language tag of the form `<2xx>` prepended to the Spanish source sentence in the `src` field, while the `tgt` field holds the corresponding sentence in the low-resource language: ```json { "src": "<2wo> El Océano Austral es el cuerpo de agua que rodea el Océano Austral.", "tgt": "Mbàmbulaan gu Bëj-saalum gi mooy mbalkaam ndox mi ne ci li wër Dottub Bëj-saalum bi." } ``` ### Data Splits The dataset contains a single split: `train`. ## Raw Dataset Composition | Type | lang_low | lang_high | Source dataset | Rows | |---|---|---|---|---:| | Parallel | ZGH | EN | commonvoice | 1,451 | | Parallel | ZGH | EN | tatoeba-collectivat | 316,374 | | Parallel | ZGH | ES | tatoeba-collectivat | 25,287 | | Parallel | ZGH | FR | tatoeba-collectivat | 49,556 | | Mono | ZGH | - | corpus-ircam | 26,791 | | Mono | ZGH | - | wajdm-collectivat | 1,002 | | Mono | ZGH | - | tamazight-wikidump | 20,342 | | **Subtotal ZGH** | | | | **440,803** | | Parallel | PS | EN | nllb | 11,348,028 | | Parallel | PS | EN | ccaligned | 299,628 | | Parallel | PS | EN | gnome | 95,312 | | Parallel | PS | EN | xlent | 54,922 | | Parallel | PS | EN | paracrawl | 26,321 | | Parallel | PS | EN | wikimedia | 3,748 | | Parallel | PS | EN | kde4 | 3,377 | | Parallel | PS | EN | tico19 | 3,071 | | Parallel | PS | ES | multiccaligned | 136,500 | | Parallel | PS | ES | gnome | 122,373 | | Parallel | PS | ES | xlent | 25,556 | | Parallel | PS | ES | multiparacrawl | 4,788 | | Parallel | PS | ES | kde4 | 3,403 | | Mono | PS | - | pashto-wikidump | 609,369 | | **Subtotal PS** | | | | **12,736,396** | | Parallel | WO | EN | ccaligned | 88,440 | | Parallel | WO | EN | nllb | 1,460,420 | | Parallel | WO | FR | multiccaligned | 24,255 | | Parallel | WO | FR | nllb | 376,455 | | Mono | WO | - | wolof-wikidump | 30,916 | | **Subtotal WO** | | | | **1,980,486** | | Parallel | ROM | EN | bible-uedin | 15,855 | | Parallel | ROM | ES | bible-uedin | 7,931 | | Mono | ROM | - | romani-wikidump | 8,687 | | Mono | ROM | - | curriculum-framework-for-romani | 1,283 | | **Subtotal ROM** | | | | **33,756** | | **TOTAL** | | | | **15,191,441** | **Parallel rows:** 14,493,051 - **Monolingual rows:** 698,390 ## Dataset Creation ### Curation Rationale The TRAIN project (Task 2.4 Collection of textual resources for the targeted languages) aimed to compile at least 300,000 sentences per language to support the development of MT systems for the migrant and minority languages of communities in Spain. Due to the extreme scarcity of bilingual corpora for these languages, the collection strategy targeted two complementary types of resources: 1. **Bilingual corpora**: the preferred type, covering any language pair involving the low-resource language and at least one high-resource partner (Spanish, English, or French as pivot). 2. **Monolingual corpora**: used to supplement bilingual resources and to generate synthetic parallel data via machine translation and backtranslation. And additionally applied: 3. **Synthetic data** generation strategies included pivot-language translation (e.g., Pashto-English → Spanish) and backtranslation from monolingual data. Synthetic data are included exclusively in the `multi-training-set.json` file. ### Source Data #### Initial Data Collection and Normalization **Bilingual source datasets:** | Dataset | Language pair | Domain / Source | Rows in this corpus | |---|---|---|---:| | CommonVoice (Col·lectivaT) | EN-ZGH | Voice localisation | 1,451 | | Tatoeba (Col·lectivaT) | EN-ZGH | Community translations | 316,374 | | Tatoeba (Col·lectivaT) | ES-ZGH | Community translations | 25,287 | | Tatoeba (Col·lectivaT) | FR-ZGH | Community translations | 49,556 | | NLLB (OPUS) | EN-PS | Web-crawled, multi-domain | 11,348,028 | | CCAligned (OPUS) | EN-PS | Web-crawled (CommonCrawl) | 299,628 | | GNOME (OPUS) | EN-PS | Software localisation | 95,312 | | XLEnt (OPUS) | EN-PS | Named entities | 54,922 | | ParaCrawl (OPUS) | EN-PS | Web-crawled | 26,321 | | Wikimedia (OPUS) | EN-PS | Wikipedia metadata | 3,748 | | KDE4 (OPUS) | EN-PS | Software localisation | 3,377 | | TICO-19 (OPUS) | EN-PS | Medical/COVID-19 | 3,071 | | MultiCCAligned (OPUS) | ES-PS | Web-crawled (CommonCrawl) | 136,500 | | GNOME (OPUS) | ES-PS | Software localisation | 122,373 | | XLEnt (OPUS) | ES-PS | Named entities | 25,556 | | MultiParaCrawl (OPUS) | ES-PS | Web-crawled | 4,788 | | KDE4 (OPUS) | ES-PS | Software localisation | 3,403 | | CCAligned (OPUS) | EN-WO | Web-crawled (CommonCrawl) | 88,440 | | NLLB (OPUS) | EN-WO | Web-crawled, multi-domain | 1,460,420 | | MultiCCAligned (OPUS) | FR-WO | Web-crawled (CommonCrawl) | 24,255 | | NLLB (OPUS) | FR-WO | Web-crawled, multi-domain | 376,455 | | Bible-UEdin (OPUS) | EN-ROM | Religious text (Bible) | 15,855 | | Bible-UEdin (OPUS) | ES-ROM | Religious text (Bible) | 7,931 | **Monolingual source datasets:** | Dataset | Language | Domain / Source | Rows in this corpus | |---|---|---|---:| | Corpus IRCAM (TALAM) | ZGH | Mixed texts (Latin + Tifinagh), converted to Tifinagh | 26,791 | | WAJDM (Col·lectivaT) | ZGH | Mixed domains | 1,002 | | Tamazight Wikipedia dump | ZGH | Encyclopaedic (Wikipedia) | 20,342 | | Pashto Wikipedia dump | PS | Encyclopaedic (Wikipedia) | 609,369 | | Wolof Wikipedia dump | WO | Encyclopaedic (Wikipedia) | 30,916 | | Romani Wikipedia dump | ROM | Encyclopaedic (Wikipedia) | 8,687 | | curriculum-framework-for-romani (Council of Europe) | ROM | Institutional/education | 1,283 | **Data Filtering and Normalization:** - The source parallel datasets from OPUS and NLLB had already been filtered with language identification [Fasttext](https://github.com/facebookresearch/fastText) prior to inclusion in this corpus. - Wikipedia dumps were cleaned and split into sentences before inclusion. - The Corpus IRCAM originally contains text in both Latin and Tifinagh scripts; the texts written in the Latin script were transliterated into Tifinagh by using a [Tifinagh transliterator](https://github.com/CollectivaT-dev/tifinagh_transliterator). - No additional filtering or deduplication has been applied during the aggregation step that produced this corpus. - An additional language identification pass with [Fasttext](https://github.com/facebookresearch/fastText) was applied to the `multi-training-set.json` dataset to remove sentence pairs where the low-resource or high-resource language was misidentified. This significantly reduced the size of the noisier web-crawled datasets. Monolingual low-resource sentences that passed the filter were machine-translated into Spanish with [Google Translate](https://translate.google.com/) to generate synthetic parallel data. Additionally, existing EN-XX and FR-XX parallel pairs were automatically translated into Spanish via pivot translation to augment the ES-XX training data, given the project's primary goal of building Spanish↔low-resource MT systems. #### Who are the source language producers? - **Tamazight:** Amazigh-speaking communities; data curated by Col·lectivaT (with funding from the Municipality of Barcelona and the Government of Catalonia), IRCAM (Institut Royal de la Culture Amazighe / TALAM), and Wikipedia contributors. - **Pashto:** Pashto-speaking communities in Afghanistan, Pakistan, and diaspora; data from OPUS/NLLB (OPUS) and Wikipedia contributors. - **Wolof:** Wolof-speaking communities in Senegal, The Gambia, and Mauritania; data from OPUS/NLLB (OPUS) and Wikipedia contributors. - **Romani:** Romani-speaking communities across Europe; data from OPUS/Bible-UEdin, Wikipedia contributors, and the Council of Europe (institutional documents). ### Annotations #### Annotation process The dataset does not contain any manual annotations beyond the parallel alignments, which were either preserved from source datasets or validated through automated alignment scoring. #### Who are the annotators? [N/A] ### Personal and Sensitive Information Given that this dataset is derived from pre-existing datasets that contain crawled data, and that no specific anonymisation process has been applied, personal and sensitive information may be present in the data. This needs to be considered when using the data for training models. ## Considerations for Using the Data ### Social Impact of Dataset This dataset contributes resources for four extremely under-resourced languages spoken by migrant and minority communities. Improved MT systems for these languages can facilitate communication with public services, support access to information, and help preserve and document these languages digitally. This is especially relevant in the context of Spain, where Romani, Tamazight-speaking Amazigh communities, and migrant communities speaking Pashto and Wolof are present. ### Discussion of Biases No specific bias mitigation strategies were applied to this dataset beyond deduplication and minimal quality filtering. Inherent biases may exist within the data, reflecting the biases present in the source datasets, which include web-crawled content, subtitles, news articles, and other user-generated or institutionally produced text. Additional content-specific considerations: - **Domain imbalance:** Pashto data is heavily dominated by the NLLB corpus (11.3M of 12.7M total PS rows), which is web-crawled. Other domains are relatively small. - **Dialectal variation:** Romani in particular has significant dialectal diversity; the data does not represent all dialects equally. Tamazight similarly covers primarily Standard Moroccan Amazigh (ZGH) rather than the full Amazigh dialect spectrum. - **Script inconsistencies:** Tamazight data may contain mixed Tifinagh/Latin representations in some source files. Additionally, the mentioned [Tifinagh transliterator](https://github.com/CollectivaT-dev/tifinagh_transliterator) is an experimental tool whose results have not been verified for accuracy or quality. ### Other Known Limitations - Reduced domain coverage for Romani (primarily Bible text and Wikipedia) and for some Tamazight pairs. - Known limitations of crawled data apply: noise, misalignments, and low-quality sentence pairs may be present, particularly in large web-crawled corpora (NLLB, CCAligned, MultiParaCrawl). - The synthetic parallel data in `multi-training-set.json` was generated by machine translation (Google Translate) and is therefore subject to the inherent limitations of MT systems, including translation errors, hallucinations, and propagation of the source model's biases. Synthetic sentences may also lack the naturalness and variability of authentic text, which can negatively affect the fluency of MT models trained on them. ## Additional Information ### Dataset Curators Machine Translation Group, AI Institute, Barcelona Supercomputing Center (ai_institute_mt@bsc.es). ### Funding This work has been supported by the Spanish project PID2021-123988OB-C33 funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, as part of the TRAIN project (*Traducción Automática para la Inclusión*), Work Package 2, Task 2.4. ### Acknowledgements The Tamazight data was collected with support of [Col·lectivaT](https://huggingface.co/collectivat). The Tamazight monolingual corpus was provided by [IRCAM](https://www.ircam.ma/) trough the portal [TALAM (Traitement Automatique de la Langue Amazighe)](https://tal.ircam.ma/talam/). Several datasets were obtained from [OPUS](https://opus.nlpl.eu/), an open collection of multilingual parallel corpora (Tiedemann, J., 2012. Parallel Data, Tools and Interfaces in OPUS. *Proceedings of LREC 2012*). Each OPUS dataset retains the licence of its original source; credit is due to the respective dataset creators and contributors. ### Licensing Information This work is licensed under a [Creative Commons Attribution ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) licence. Note: Individual source datasets carry their own licences. Users of this aggregated corpus should verify licence compatibility with the intended use case. Users must ensure compliance with all applicable third-party licenses. ### Citation Information [N/A] ### Contributions [N/A]

This corpus was compiled as part of the TRAIN project (Traducción Automática para la Inclusión, Automatic Translation for Inclusion), funded by MCIN/AEI and ERDF. It aggregates 15,191,441 sentence-level entries covering four extremely low-resource languages: Tamazight/Amazigh (ZGH), Pashto (PS), Wolof (WO), and Romani (ROM), paired with one or more high-resource counterparts (English, Spanish, French), plus monolingual data for each low-resource language. The dataset is intended primarily for training machine translation (MT) systems targeting under-resourced languages of migrant and minority communities in Spain. It consolidates 30 source datasets drawn from OPUS, the NLLB collection, CommonVoice, Tatoeba, Col·lectivaT, IRCAM, and other sources into a single unified large dataset (see Source Data section for the full list). The data spans a wide range of domains including web-crawled text, software localisation, news, encyclopaedic content (Wikipedia), religious text (Bible), educational contents, and institutional documents.

提供机构:
BSC-LT
搜集汇总
数据集介绍
BSC-LT/MULTI_corpus 数据集图片
构建方式
面对塔马齐格特语、普什图语、沃洛夫语和罗姆语等四种极度低资源语言在机器翻译领域长期面临语料匮乏的困境,MULTI-Corpus的构建采取多源聚合与合成增强相结合的策略。该语料库整合了来自OPUS、NLLB、CommonVoice、Tatoeba、Col·lectivaT及IRCAM等30个来源的数据,汇聚成15,191,441条句级条目。原始数据保留平行句对与单语文本,并针对塔马齐格特语拉丁转提非纳文进行转写,维基百科文本经清洗分句后纳入。训练版本则经Fasttext语言识别过滤,利用谷歌翻译将低资源单语数据译为西班牙语,并通过英语、法语枢轴翻译扩充西班牙语-低资源语言平行数据。
使用方法
使用该数据集时,研究者可根据需求选择不同版本。原始TSV文件适用于探索性分析、自定义过滤流程,或作为训练数据的上游来源;预处理后的JSON训练集则按MADLAD-400模型输入格式组织,源字段中的西班牙语句子前需附加形如<2xx>的目标语言标签,目标字段存放低资源语言句子。加载数据后,可直接用于训练西班牙语与四种低资源语言间的机器翻译系统,或用于低资源语言的跨语言迁移研究。使用时应留意合成数据可能引入的翻译错误与偏见,并核查各源数据集许可证兼容性。
背景与挑战
背景概述
为应对西班牙境内移民与少数族群语言在公共服务及信息获取中的交流壁垒,巴塞罗那超级计算中心人工智能研究所机器翻译团队于TRAIN项目(PID2021-123988OB-C33)框架下构建了MULTI-Corpus。该语料库汇集逾1519万句对,覆盖塔马齐格特语、普什图语、沃洛夫语和罗姆语四种极度低资源语言,整合OPUS、NLLB、CommonVoice等30个来源,旨在为西班牙语与低资源语言间的机器翻译系统提供大规模训练资源。其发布显著缓解了上述语言在自然语言处理领域长期面临的数据匮乏困境,对促进语言包容性技术发展具有里程碑意义。
当前挑战
该数据集所应对的核心领域难题在于,四种目标语言均属极度低资源范畴,双语平行数据稀缺且零散,同时伴随显著的方言分歧、正字法不统一及右至左书写等复杂语言现象。构建过程中,海量网络爬取数据引入严重噪声与错位,需依赖语言识别工具进行过滤,但过滤本身可能误删有效样本;塔马齐格特语的拉丁与提非纳文混合文本依赖实验性转写工具,其准确性未经充分验证;而借助谷歌翻译生成的合成数据虽扩充了规模,却不可避免地继承了机器翻译的误差、幻觉及源模型偏见,致使数据质量与领域覆盖的均衡性面临持续挑战。
常用场景
经典使用场景
在机器翻译领域,针对极低资源语言缺乏大规模平行语料的困境,MULTI-Corpus 数据集凭借其涵盖塔马齐格特语、普什图语、沃洛夫语和罗姆语四种弱势语言与英语、西班牙语、法语等高资源语言配对的一千五百万余句对,成为训练神经机器翻译模型的经典语料来源。研究者常将其用于构建西班牙语与上述低资源语言之间的双向翻译系统,或利用其单语数据开展回译增强,从而在数据稀缺条件下探索迁移学习与多语言联合训练的效能边界。
解决学术问题
该数据集有力回应了低资源语言机器翻译研究中平行语料匮乏与领域覆盖不均的共性难题。通过整合 OPUS、NLLB、CommonVoice 等三十个来源,它提供了跨网络爬取、软件本地化、新闻、百科及宗教文本的多域句对,使研究者得以系统考察领域偏移、方言变异和文字转写对翻译质量的影响,并为语言识别、数据过滤及合成数据生成等预处理策略的评估提供了可复现的基准。
实际应用
在现实应用中,MULTI-Corpus 服务于西班牙移民与少数族裔社区的公共服务沟通需求,其训练出的翻译系统可协助医疗、政务及教育场景下的跨语言信息获取。此外,该数据集亦可用于语言文档化与数字化保护工作,为塔马齐格特语、普什图语、沃洛夫语和罗姆语的在线内容生成、社区翻译工具开发及语言学习资源建设提供基础支撑。
数据集最近研究
最新研究方向
在低资源机器翻译领域,面向移民与少数族裔语言的数据资源构建已成为近年来的研究热点。MULTI-Corpus的推出,整合了塔马齐格特语、普什图语、沃洛夫语和罗姆语四种极低资源语言与英语、西班牙语、法语的高资源对照,共计逾1500万句对,为神经机器翻译模型的训练提供了关键支撑。该数据集不仅涵盖了网络爬取、软件本地化、新闻、百科及宗教文本等多领域语料,还通过合成数据生成策略进一步扩充了训练集,显著缓解了低资源语言对稀缺的困境。其前沿研究方向包括基于枢轴语言的翻译增强、回译技术优化以及语言识别过滤对数据质量的提升。该数据集的发布对于促进西班牙等地区移民与少数族裔的社会融合、语言保护及数字包容具有深远影响,同时为低资源机器翻译的公平性与可及性研究树立了重要里程碑。
以上内容由遇见数据集搜集并总结生成
二维码
社区交流群
二维码
科研交流群
商业服务