MULTI_corpus

# Dataset Card for MULTI-Corpus ## Dataset Description - **Point of Contact:** [e-mail] ### Dataset Summary This corpus was compiled as part of the TRAIN project (*Traducción Automática para la Inclusión*, Automatic Translation for Inclusion), funded by MCIN/AEI and ERDF. It aggregates **15,191,441 sentence-level entries** covering four extremely low-resource languages: Tamazight/Amazigh (ZGH), Pashto (PS), Wolof (WO), and Romani (ROM), paired with one or more high-resource counterparts (English, Spanish, French), plus monolingual data for each low-resource language. The dataset is intended primarily for training machine translation (MT) systems targeting under-resourced languages of migrant and minority communities in Spain. It consolidates 30 source datasets drawn from OPUS, the NLLB collection, CommonVoice, Tatoeba, Col·lectivaT, IRCAM, and other sources into a single unified large dataset (see Source Data section for the full list). The data spans a wide range of domains including web-crawled text, software localisation, news, encyclopaedic content (Wikipedia), religious text (Bible), educational contents, and institutional documents. ### Supported Tasks and Leaderboards - **Machine Translation:** parallel sentence pairs for the language pairs EN-ZGH, ES-ZGH, FR-ZGH, EN-PS, ES-PS, EN-WO, FR-WO, EN-ROM, ES-ROM. ### Languages | Language | ISO code | Script | Notes | |---|---|---|---| | Tamazight (Standard Moroccan Amazigh) | `zgh` | Tifinagh (Unicode U+2D30-U+2D7F) | Recently standardised; inconsistent orthographic conventions exist across dialects | | Pashto | `ps` | Pashto/Perso-Arabic (right-to-left) | Variant of the Arabic alphabet adapted for Pashto-specific sounds | | Wolof | `wo` | Latin | Standard Latin alphabet adapted during the colonial period | | Romani | `rom` | Latin with diacritics | Diacritical marks added to represent sounds absent from standard Latin languages; significant dialectal variation | | English | `en` | Latin | High-resource parallel counterpart | | Spanish | `es` | Latin | High-resource parallel counterpart | | French | `fr` | Latin | High-resource parallel counterpart | ## Dataset Structure This dataset is provided in two different versions: The `multi-corpus-raw.tsv` file is the unfiltered aggregation of all 30 source datasets listed in the Source Data section. It contains only **original (non-synthetic)** data, covering original parallel pairs (14,493,051 rows) and monolingual sentences (698,390 rows) across the four low-resource languages ZGH, PS, WO, and ROM, paired with EN, ES, and FR. No filtering, deduplication, or synthetic data generation has been applied; the file preserves the data exactly as found in the source corpora. It is suited for exploratory analysis, custom filtering pipelines, and as the upstream source for the training set. The `multi-training-set.json` file is a preprocessed, filtered, and augmented training-ready dataset derived from the raw corpus. It contains Spanish (ES) sentences paired with sentences in ZGH, PS, WO, and ROM, and was produced through three steps: (1) filtering the raw corpus with the language identification tool [Fasttext](https://github.com/facebookresearch/fastText), which significantly reduced the size of the noisier web-crawled datasets; (2) generating synthetic ES-XX parallel data by automatically translating monolingual low-resource sentences into Spanish using [Google Translate](https://translate.google.com/); and (3) augmenting the ES-XX parallel data by pivot-translating existing EN-XX and FR-XX sentence pairs into Spanish. ### Data Instances The dataset `multi-corpus-raw.tsv` is provided in TSV (tab-separated values) format. It contains one sentence per row with the following structure: ```json { "type": "Parallel", "lang_low": "ZGH", "lang_high": "EN", "dataset": "commonvoice", "text_low": "ⴰⵔⴳⴰⵜ ⵙ ⵓⴳⵕⴹ ⵏⵏⵓⵏ", "text_high": "Contribute Your Voice" } ``` Example monolingual row: ```json { "type": "Mono", "lang_low": "ROM", "lang_high": "-", "dataset": "curriculum-framework-for-romani", "text_low": "I Romani čhib si po drom te xasavol, te na avel la protekcija thaj ažutipe lovenca katar le nacionalni thaj internacionalni aktorura.", "text_high": "-" } ``` ### Data Fields - `type`: `Parallel` (bilingual sentence pair) or `Mono` (monolingual sentence). - `lang_low`: ISO code of the low-resource language (`ZGH`, `PS`, `WO`, `ROM`). - `lang_high`: ISO code of the high-resource paired language (`EN`, `ES`, `FR`), or `-` for monolingual entries. - `dataset`: Name of the source dataset (see Source Data section for the full list). - `text_low`: Sentence in the low-resource language. - `text_high`: Paired sentence in the high-resource language, or `-` for monolingual entries. The file `multi-training-set.json` is provided in JSON format. The structure follows the input format required by [MADLAD-400](https://huggingface.co/google/madlad400-3b-mt), the multilingual MT model used for experimentation in the TRAIN project. Specifically, the target low-resource language is indicated by a language tag of the form `<2xx>` prepended to the Spanish source sentence in the `src` field, while the `tgt` field holds the corresponding sentence in the low-resource language: ```json { "src": "<2wo> El Océano Austral es el cuerpo de agua que rodea el Océano Austral.", "tgt": "Mbàmbulaan gu Bëj-saalum gi mooy mbalkaam ndox mi ne ci li wër Dottub Bëj-saalum bi." } ``` ### Data Splits The dataset contains a single split: `train`. ## Raw Dataset Composition | Type | lang_low | lang_high | Source dataset | Rows | |---|---|---|---|---:| | Parallel | ZGH | EN | commonvoice | 1,451 | | Parallel | ZGH | EN | tatoeba-collectivat | 316,374 | | Parallel | ZGH | ES | tatoeba-collectivat | 25,287 | | Parallel | ZGH | FR | tatoeba-collectivat | 49,556 | | Mono | ZGH | - | corpus-ircam | 26,791 | | Mono | ZGH | - | wajdm-collectivat | 1,002 | | Mono | ZGH | - | tamazight-wikidump | 20,342 | | **Subtotal ZGH** | | | | **440,803** | | Parallel | PS | EN | nllb | 11,348,028 | | Parallel | PS | EN | ccaligned | 299,628 | | Parallel | PS | EN | gnome | 95,312 | | Parallel | PS | EN | xlent | 54,922 | | Parallel | PS | EN | paracrawl | 26,321 | | Parallel | PS | EN | wikimedia | 3,748 | | Parallel | PS | EN | kde4 | 3,377 | | Parallel | PS | EN | tico19 | 3,071 | | Parallel | PS | ES | multiccaligned | 136,500 | | Parallel | PS | ES | gnome | 122,373 | | Parallel | PS | ES | xlent | 25,556 | | Parallel | PS | ES | multiparacrawl | 4,788 | | Parallel | PS | ES | kde4 | 3,403 | | Mono | PS | - | pashto-wikidump | 609,369 | | **Subtotal PS** | | | | **12,736,396** | | Parallel | WO | EN | ccaligned | 88,440 | | Parallel | WO | EN | nllb | 1,460,420 | | Parallel | WO | FR | multiccaligned | 24,255 | | Parallel | WO | FR | nllb | 376,455 | | Mono | WO | - | wolof-wikidump | 30,916 | | **Subtotal WO** | | | | **1,980,486** | | Parallel | ROM | EN | bible-uedin | 15,855 | | Parallel | ROM | ES | bible-uedin | 7,931 | | Mono | ROM | - | romani-wikidump | 8,687 | | Mono | ROM | - | curriculum-framework-for-romani | 1,283 | | **Subtotal ROM** | | | | **33,756** | | **TOTAL** | | | | **15,191,441** | **Parallel rows:** 14,493,051 - **Monolingual rows:** 698,390 ## Dataset Creation ### Curation Rationale The TRAIN project (Task 2.4 Collection of textual resources for the targeted languages) aimed to compile at least 300,000 sentences per language to support the development of MT systems for the migrant and minority languages of communities in Spain. Due to the extreme scarcity of bilingual corpora for these languages, the collection strategy targeted two complementary types of resources: 1. **Bilingual corpora**: the preferred type, covering any language pair involving the low-resource language and at least one high-resource partner (Spanish, English, or French as pivot). 2. **Monolingual corpora**: used to supplement bilingual resources and to generate synthetic parallel data via machine translation and backtranslation. And additionally applied: 3. **Synthetic data** generation strategies included pivot-language translation (e.g., Pashto-English → Spanish) and backtranslation from monolingual data. Synthetic data are included exclusively in the `multi-training-set.json` file. ### Source Data #### Initial Data Collection and Normalization **Bilingual source datasets:** | Dataset | Language pair | Domain / Source | Rows in this corpus | |---|---|---|---:| | CommonVoice (Col·lectivaT) | EN-ZGH | Voice localisation | 1,451 | | Tatoeba (Col·lectivaT) | EN-ZGH | Community translations | 316,374 | | Tatoeba (Col·lectivaT) | ES-ZGH | Community translations | 25,287 | | Tatoeba (Col·lectivaT) | FR-ZGH | Community translations | 49,556 | | NLLB (OPUS) | EN-PS | Web-crawled, multi-domain | 11,348,028 | | CCAligned (OPUS) | EN-PS | Web-crawled (CommonCrawl) | 299,628 | | GNOME (OPUS) | EN-PS | Software localisation | 95,312 | | XLEnt (OPUS) | EN-PS | Named entities | 54,922 | | ParaCrawl (OPUS) | EN-PS | Web-crawled | 26,321 | | Wikimedia (OPUS) | EN-PS | Wikipedia metadata | 3,748 | | KDE4 (OPUS) | EN-PS | Software localisation | 3,377 | | TICO-19 (OPUS) | EN-PS | Medical/COVID-19 | 3,071 | | MultiCCAligned (OPUS) | ES-PS | Web-crawled (CommonCrawl) | 136,500 | | GNOME (OPUS) | ES-PS | Software localisation | 122,373 | | XLEnt (OPUS) | ES-PS | Named entities | 25,556 | | MultiParaCrawl (OPUS) | ES-PS | Web-crawled | 4,788 | | KDE4 (OPUS) | ES-PS | Software localisation | 3,403 | | CCAligned (OPUS) | EN-WO | Web-crawled (CommonCrawl) | 88,440 | | NLLB (OPUS) | EN-WO | Web-crawled, multi-domain | 1,460,420 | | MultiCCAligned (OPUS) | FR-WO | Web-crawled (CommonCrawl) | 24,255 | | NLLB (OPUS) | FR-WO | Web-crawled, multi-domain | 376,455 | | Bible-UEdin (OPUS) | EN-ROM | Religious text (Bible) | 15,855 | | Bible-UEdin (OPUS) | ES-ROM | Religious text (Bible) | 7,931 | **Monolingual source datasets:** | Dataset | Language | Domain / Source | Rows in this corpus | |---|---|---|---:| | Corpus IRCAM (TALAM) | ZGH | Mixed texts (Latin + Tifinagh), converted to Tifinagh | 26,791 | | WAJDM (Col·lectivaT) | ZGH | Mixed domains | 1,002 | | Tamazight Wikipedia dump | ZGH | Encyclopaedic (Wikipedia) | 20,342 | | Pashto Wikipedia dump | PS | Encyclopaedic (Wikipedia) | 609,369 | | Wolof Wikipedia dump | WO | Encyclopaedic (Wikipedia) | 30,916 | | Romani Wikipedia dump | ROM | Encyclopaedic (Wikipedia) | 8,687 | | curriculum-framework-for-romani (Council of Europe) | ROM | Institutional/education | 1,283 | **Data Filtering and Normalization:** - The source parallel datasets from OPUS and NLLB had already been filtered with language identification [Fasttext](https://github.com/facebookresearch/fastText) prior to inclusion in this corpus. - Wikipedia dumps were cleaned and split into sentences before inclusion. - The Corpus IRCAM originally contains text in both Latin and Tifinagh scripts; the texts written in the Latin script were transliterated into Tifinagh by using a [Tifinagh transliterator](https://github.com/CollectivaT-dev/tifinagh_transliterator). - No additional filtering or deduplication has been applied during the aggregation step that produced this corpus. - An additional language identification pass with [Fasttext](https://github.com/facebookresearch/fastText) was applied to the `multi-training-set.json` dataset to remove sentence pairs where the low-resource or high-resource language was misidentified. This significantly reduced the size of the noisier web-crawled datasets. Monolingual low-resource sentences that passed the filter were machine-translated into Spanish with [Google Translate](https://translate.google.com/) to generate synthetic parallel data. Additionally, existing EN-XX and FR-XX parallel pairs were automatically translated into Spanish via pivot translation to augment the ES-XX training data, given the project's primary goal of building Spanish↔low-resource MT systems. #### Who are the source language producers? - **Tamazight:** Amazigh-speaking communities; data curated by Col·lectivaT (with funding from the Municipality of Barcelona and the Government of Catalonia), IRCAM (Institut Royal de la Culture Amazighe / TALAM), and Wikipedia contributors. - **Pashto:** Pashto-speaking communities in Afghanistan, Pakistan, and diaspora; data from OPUS/NLLB (OPUS) and Wikipedia contributors. - **Wolof:** Wolof-speaking communities in Senegal, The Gambia, and Mauritania; data from OPUS/NLLB (OPUS) and Wikipedia contributors. - **Romani:** Romani-speaking communities across Europe; data from OPUS/Bible-UEdin, Wikipedia contributors, and the Council of Europe (institutional documents). ### Annotations #### Annotation process The dataset does not contain any manual annotations beyond the parallel alignments, which were either preserved from source datasets or validated through automated alignment scoring. #### Who are the annotators? [N/A] ### Personal and Sensitive Information Given that this dataset is derived from pre-existing datasets that contain crawled data, and that no specific anonymisation process has been applied, personal and sensitive information may be present in the data. This needs to be considered when using the data for training models. ## Considerations for Using the Data ### Social Impact of Dataset This dataset contributes resources for four extremely under-resourced languages spoken by migrant and minority communities. Improved MT systems for these languages can facilitate communication with public services, support access to information, and help preserve and document these languages digitally. This is especially relevant in the context of Spain, where Romani, Tamazight-speaking Amazigh communities, and migrant communities speaking Pashto and Wolof are present. ### Discussion of Biases No specific bias mitigation strategies were applied to this dataset beyond deduplication and minimal quality filtering. Inherent biases may exist within the data, reflecting the biases present in the source datasets, which include web-crawled content, subtitles, news articles, and other user-generated or institutionally produced text. Additional content-specific considerations: - **Domain imbalance:** Pashto data is heavily dominated by the NLLB corpus (11.3M of 12.7M total PS rows), which is web-crawled. Other domains are relatively small. - **Dialectal variation:** Romani in particular has significant dialectal diversity; the data does not represent all dialects equally. Tamazight similarly covers primarily Standard Moroccan Amazigh (ZGH) rather than the full Amazigh dialect spectrum. - **Script inconsistencies:** Tamazight data may contain mixed Tifinagh/Latin representations in some source files. Additionally, the mentioned [Tifinagh transliterator](https://github.com/CollectivaT-dev/tifinagh_transliterator) is an experimental tool whose results have not been verified for accuracy or quality. ### Other Known Limitations - Reduced domain coverage for Romani (primarily Bible text and Wikipedia) and for some Tamazight pairs. - Known limitations of crawled data apply: noise, misalignments, and low-quality sentence pairs may be present, particularly in large web-crawled corpora (NLLB, CCAligned, MultiParaCrawl). - The synthetic parallel data in `multi-training-set.json` was generated by machine translation (Google Translate) and is therefore subject to the inherent limitations of MT systems, including translation errors, hallucinations, and propagation of the source model's biases. Synthetic sentences may also lack the naturalness and variability of authentic text, which can negatively affect the fluency of MT models trained on them. ## Additional Information ### Dataset Curators Machine Translation Group, AI Institute, Barcelona Supercomputing Center ([e-mail]). ### Funding This work has been supported by the Spanish project PID2021-123988OB-C33 funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, as part of the TRAIN project (*Traducción Automática para la Inclusión*), Work Package 2, Task 2.4. ### Acknowledgements The Tamazight data was collected with support of [Col·lectivaT](https://huggingface.co/collectivat). The Tamazight monolingual corpus was provided by [IRCAM](https://www.ircam.ma/) trough the portal [TALAM (Traitement Automatique de la Langue Amazighe)](https://tal.ircam.ma/talam/). Several datasets were obtained from [OPUS](https://opus.nlpl.eu/), an open collection of multilingual parallel corpora (Tiedemann, J., 2012. Parallel Data, Tools and Interfaces in OPUS. *Proceedings of LREC 2012*). Each OPUS dataset retains the licence of its original source; credit is due to the respective dataset creators and contributors. ### Licensing Information This work is licensed under a [Creative Commons Attribution ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) licence. Note: Individual source datasets carry their own licences. Users of this aggregated corpus should verify licence compatibility with the intended use case. Users must ensure compliance with all applicable third-party licenses. ### Citation Information [N/A] ### Contributions [N/A]