BSC_ParaMT_8

# Dataset Card for BSC_ParaMT_8 ## Dataset Description - **Point of Contact:** [e-mail] ### Dataset Summary Large-scale multilingual parallel corpus covering Catalan, Spanish, and English paired with Arabic, Hindi, Chinese, Japanese, and Korean. Built by aggregating and carefully filtering multiple public sources, it provides sentence-level alignments for training Machine Translation systems. The Spanish portion of the dataset includes synthetic data generated by translating original English sentences into Spanish. Similarly, the Catalan portion includes synthetic data generated by translating original English and Spanish sentences into Catalan. The synthetic text was produced by using our model [SalamandraTA 7B Instruct](https://huggingface.co/BSC-LT/salamandra-7b-instruct). ### Supported Tasks and Leaderboards The dataset is primarily designed for Machine Translation pairing Catalan, Spanish, and English with Arabic, Hindi, Chinese, Japanese, and Korean. Typical uses include supervised MT training, fine-tuning multilingual models, and data augmentation. ### Languages The dataset includes parallel sentences across the following language pairs: ### CA-XX pairs | Language pair | Codes | Size (sentences) |-------------------|-------|------------------ | Catalan-Arabic | ca-ar | 72,339,999 | Catalan-Hindi | ca-hi | 32,783,148 | Catalan-Chinese | ca-zh | 94,187,858 | Catalan-Japanese | ca-ja | 27,826,529 | Catalan-Korean | ca-ko | 36,888,994 | **Total** | | **264,026,528** ### ES-XX pairs | Language pair | Codes | Size (sentences) |-------------------|-------|------------------ | Spanish-Arabic | es-ar | 48,830,472 | Spanish-Hindi | es-hi | 33,703,968 | Spanish-Chinese | es-zh | 78,828,366 | Spanish-Japanese | es-ja | 32,129,479 | Spanish-Korean | es-ko | 36,719,824 | **Total** | | **230,212,109** ### EN-XX pairs | Language pair | Codes | Size (sentences) |-------------------|-------|------------------ | English-Arabic | en-ar | 72,842,230 | English-Hindi | en-hi | 27,256,919 | English-Chinese | en-zh | 65,047,374 | English-Japanese | en-ja | 29,017,543 | English-Korean | en-ko | 29,407,174 | **Total** | | **223,571,240** **Grand Total: 717,809,877** ## Dataset Structure ### Data Instances The dataset is provided in parquet format, with a separate file for each language pair. For some language pairs, several parquet files containing partial datasets are provided. Each row contains a parallel sentence pair with the following structure: ```json { "l1_sentence": "Cada nit observava les estrelles des de la finestra de la seva habitació i desitjava tenir-ne una.", "l2_sentence": " 毎晩、少年は窓から星を見て、彼には自分のものをひとつ持っていたい願望があった。", "l1": "ca", "l2": "ja" } ``` ### Data Fields - `l1_sentence`: The sentence in the first language (string) - `l2_sentence`: The parallel sentence in the second language (string) - `l1`: ISO 639-1 code of the first language (string) - `l2`: ISO 639-1 code of the second language (string) ### Data Splits The dataset contains a single split: `train`. ## Dataset Creation ### Curation Rationale This dataset is aimed at promoting the development of Machine Translation pairing Catalan, Spanish, and English with Arabic, Hindi, Chinese, Japanese, and Korean. It promotes research in multilingual NLP and facilitates the development of translation systems for diverse language pairs. ### Source Data #### Initial Data Collection and Normalization The corpus is a combination of the following original datasets collected from various public sources: - **[Tatoeba](https://opus.nlpl.eu/datasets/Tatoeba)**: Community-contributed sentences translations - **[UNPC](https://www.un.org/dgacm/en/content/uncorpus)**: United Nations Parallel Corpus - **[NLLB](https://opus.nlpl.eu/datasets/NLLB)**: No Language Left Behind corpus <!-- - **[MultiUN]()**: Multilingual United Nations parallel corpus --> - **[WikiMatrix](https://opus.nlpl.eu/datasets/WikiMatrix)**: Parallel sentences extracted from Wikipedia - **[News-Commentary](https://opus.nlpl.eu/datasets/News-Commentary)**: Parallel news commentary corpus - **[CCAligned](https://opus.nlpl.eu/datasets/CCAligned)**: Web-crawled parallel sentences - **[MultiCCAligned](https://opus.nlpl.eu/datasets/MultiCCAligned)**: Multilingual version of CCAligned - **[HPLT](https://hplt-project.org/)**: High Performance Language Technologies project data - **[MultiHPLT](https://hplt-project.org/)**: Multilingual HPLT corpus - **[OpenSubtitles](https://opus.nlpl.eu/datasets/OpenSubtitles)**: Community-contributed parallel subtitles from movies and TV shows - **[Anuvaad](https://opus.nlpl.eu/datasets/Anuvaad)**: Indian languages parallel corpus - **[GlobalVoices](https://opus.nlpl.eu/datasets/GlobalVoices)**: Multilingual news corpus - **[KDE4](https://inventory.clarin.gr/corpus/739)**: KDE4 localization files corpus - **[EUbookshop](https://opus.nlpl.eu/datasets/EUbookshop)**: European Union bookshop publication corpus - **[Ubuntu](https://opus.nlpl.eu/datasets/Ubuntu)**: Ubuntu localization files corpus - **[XLEnt](https://data.statmt.org/xlent/)**: Cross-lingual named entity corpus - **[Bible-UEdin](https://opus.nlpl.eu/datasets/bible-uedin)**: Bible translations from University of Edinburgh - **[Infopankki](https://opus.nlpl.eu/datasets/infopankki)**: Finnish immigration service multilingual corpus - **[TICO-19](https://opus.nlpl.eu/datasets/tico-19)**: Translation Initiative for COVID-19 corpus - **[TLDR Pages](https://opus.nlpl.eu/datasets/tldr-pages)**: Community-maintained software help pages - **[PHP](https://opus.nlpl.eu/datasets/PHP)**: PHP documentation corpus **Synthetic Data Generation:** For all CA-XX pairs and for some of the ES-XX pairs, synthetic parallel data was created by translating out of English (or Spanish). This synthetic text was produced by using our model [SalamandraTA 7B Instruct](https://huggingface.co/BSC-LT/salamandra-7b-instruct). **Data Filtering and Normalization:** Data processing was handled through our Translation Data Processing Pipeline [ParaCLEAN](https://github.com/langtech-bsc/ParaCLEAN), which includes the following processing steps: - Input handling: Read corpora in plain text (.txt), tab-separated (.tsv) or translation memory (.tmx) formats. - Compute embeddings: Compute multilingual sentence embeddings (default: [LaBSE](https://huggingface.co/sentence-transformers/LaBSE)). - Language identification: Depending on the specific language pair, language probability is calculated using [GlotLID](https://github.com/cisnlp/GlotLID) or [Lingua.py](https://github.com/pemistahl/lingua-py). - Filtering: Filter by user-defined embedding scores and language probability thresholds. - Deduplication: Remove duplicate sentence pairs and fuzzy matches across corpora. <!-- - Optional [Bifixer](https://github.com/bitextor/bifixer) cleaning: Enables optional Bifixer-based processing if Bifixer is installed. --> - Normalisation: Standardise punctuation, spacing, and casing. #### Who are the source language producers? - [OPUS NLP](https://opus.nlpl.eu/) and its constituent corpora - [NLLB Team](https://ai.meta.com/research/no-language-left-behind/) - [ParaCrawl](https://paracrawl.eu/) - [Tatoeba Community](https://tatoeba.org/) - [StatMT](https://data.statmt.org/) - [CLARIN](https://www.clarin.eu/) - [European Language Grid](https://live.european-language-grid.eu/) - Various other open-source and institutional contributors ### Annotations #### Annotation process The dataset does not contain any manual annotations beyond the parallel alignments, which were either preserved from source datasets or validated through automated alignment scoring. #### Who are the annotators? [N/A] ### Personal and Sensitive Information This dataset is derived from pre-existing datasets that may include crawled or scraped web data. By construction, it is not intended to contain personal data. However, no technical means can guarantee the complete absence of personal data at scale. This limitation is inherent to large-scale text corpora. Users are therefore responsible for assessing and managing any risks associated with the use of this data, including for model training or any other downstream application. If you believe this dataset contains personal data — or data that may enable re-identification — relating to you or a third party, you may submit a removal request by contacting us at: [e-mail]. Please include sufficient detail to locate the relevant content. Confirmed personal data will be removed in the next dataset revision, and a changelog entry will be made noting the removal. ## Considerations for Using the Data ### Social Impact of Dataset By providing this large-scale multilingual resource, we intend to promote multilingual NLP research and improve the accessibility of machine translation for the included languages, particularly for language pairs that may be underrepresented in existing resources. This contributes to reducing language barriers and supporting linguistic diversity in NLP applications. ### Discussion of Biases No specific bias mitigation strategies were applied to this dataset beyond quality filtering. Inherent biases may exist within the data, reflecting the biases present in the source datasets, which include web-crawled content, subtitles, news articles, and other user-generated or institutionally produced text. Users should be aware that the dataset contains synthetically generated text, which may reflect biases present in the translation model used. ### Other Known Limitations The dataset contains data of a general domain. Application of this dataset in more specific domains such as biomedical, legal, or other specialized fields would be of limited use. ## Additional Information ### Dataset Curators Machine Translation Group, AI Institute, the Barcelona Supercomputing Center ([e-mail]). ### Funding This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. This work has been promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/). ### Licensing Information This work is licensed under a [Open Data Commons Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1-0/). ### Citation Information [N/A] ### Contributions [N/A]