projecte-aina/CA-DE_Parallel_Corpus
收藏资源简介：
--- language: - ca - de multilinguality: - multilingual pretty_name: CA-DE Parallel Corpus size_categories: - 1M<n<10M task_categories: - translation task_ids: [] license: cc-by-sa-4.0 --- # Dataset Card for CA-DE Parallel Corpus ## Dataset Description - **Point of Contact:** langtech@bsc.es ### Dataset Summary The CA-DE Parallel Corpus is a Catalan-German dataset of parallel sentences created to support Catalan in NLP tasks, specifically Machine Translation. ### Supported Tasks and Leaderboards The dataset can be used to train Bilingual Machine Translation models between German and Catalan in any direction, as well as Multilingual Machine Translation models. ### Languages The sentences included in the dataset are in Catalan (CA) and German (DE). ## Dataset Structure ### Data Instances Two separate txt files are provided with the sentences sorted in the same order: - ca-de_6M.ca - ca-de_6M.de The dataset is additionally provided in parquet format: ca-de_6M.parquet. The parquet file contains two columns of parallel text obtained from the two original text files. Each row in the file represents a pair of parallel sentences in the two languages of the dataset. ### Data Fields [N/A] ### Data Splits The dataset contains a single split: `train`. ## Dataset Creation ### Curation Rationale This dataset is aimed at promoting the development of Machine Translation between Catalan and other languages, specifically German. ### Source Data #### Initial Data Collection and Normalization The corpus is a combination of the following original datasets collected from [Opus](https://opus.nlpl.eu/): MultiCCAligned, WikiMatrix, GNOME, KDE4, OpenSubtitles, GlobalVoices, Tatoeba, and synthetic parallel data generated from the original Spanish-Catalan Europarl and Tilde corpora made public by [SoftCatalà](https://github.com/Softcatala/Europarl-catalan). All data was filtered according to two specific criteria: - Alignment: sentence level alignments were calculated using [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) and sentence pairs with a score below 0.75 were discarded. - Language identification: the probability of being the target language was calculated using [Lingua.py](https://github.com/pemistahl/lingua-py) and sentences with a language probability score below 0.5 were discarded. The filtered datasets are then concatenated and deduplicated to form the final corpus. #### Who are the source language producers? [Opus](https://opus.nlpl.eu/) [SoftCatalà](https://github.com/Softcatala/Europarl-catalan) ### Annotations #### Annotation process The dataset does not contain any annotations. #### Who are the annotators? [N/A] ### Personal and Sensitive Information Given that this dataset is partly derived from pre-existing datasets that may contain crawled data, and that no specific anonymisation process has been applied, personal and sensitive information may be present in the data. This needs to be considered when using the data for training models. ## Considerations for Using the Data ### Social Impact of Dataset By providing this resource, we intend to promote the use of Catalan across NLP tasks, thereby improving the accessibility and visibility of the Catalan language. ### Discussion of Biases No specific bias mitigation strategies were applied to this dataset. Inherent biases may exist within the data. ### Other Known Limitations The dataset contains data of a general domain. Application of this dataset in more specific domains such as biomedical, legal etc. would be of limited use. ## Additional Information ### Dataset Curators Language Technologies Unit at the Barcelona Supercomputing Center (langtech@bsc.es). This work has been promoted and financed by the Generalitat de Catalunya through the [Aina project](https://projecteaina.cat/). ### Licensing Information This work is licensed under an [Attribution-Share Alike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). ### Citation Information [N/A] ### Contributions [N/A]
CA-DE平行语料库是一个包含9,530,709对加泰罗尼亚语（CA）和德语（DE）平行句子的数据集，旨在支持加泰罗尼亚语在自然语言处理任务中的应用，特别是机器翻译。数据集由多个来源的数据集组合而成，包括Multi CCAligned、WikiMatrix、GNOME、KDE4、OpenSubtitles、GlobalVoices、Tatoeba、Books、Europarl和Tilde。部分数据是通过从西班牙语-德语语料库中随机采样并使用PlanTL es-ca模型翻译成加泰罗尼亚语生成的。数据集经过去重和过滤，确保句子对的余弦相似度不低于0.75。数据集仅包含训练集，未包含任何注释。数据集的创建旨在促进加泰罗尼亚语在NLP任务中的使用，提高其可访问性和可见性。
CA-DE Parallel Corpus 数据集概述
数据集描述
数据集概要
CA-DE Parallel Corpus 是一个包含 9,530,709 句平行语料的加泰罗尼亚语-德语数据集。该数据集旨在支持加泰罗尼亚语在自然语言处理任务中的应用，特别是机器翻译。
支持的任务和排行榜
该数据集可用于训练德语和加泰罗尼亚语之间的双语机器翻译模型，以及多语种机器翻译模型。
语言
数据集中的句子包含加泰罗尼亚语（CA）和德语（DE）。
数据集结构
数据实例
提供了两个单独的 txt 文件，句子按相同顺序排列：
ca-de_all_2023_09_11.ca：包含 9,530,709 句加泰罗尼亚语句子。ca-de_all_2023_09_11.de：包含 9,530,709 句德语句子。
数据字段
[N/A]
数据分割
数据集包含一个分割：train。
数据集创建
策划理由
该数据集旨在促进加泰罗尼亚语与其他语言（特别是德语）之间的机器翻译开发。
源数据
初始数据收集和规范化
数据集是以下原始数据集的组合：
| 数据集 | 句子数量 |
|---|---|
| Multi CCAligned | 1,027,481 |
| WikiMatrix | 125,811 |
| GNOME | 1,241 |
| KDE4 | 105,098 |
| OpenSubtitles | 171,376 |
| GlobalVoices | 3,578 |
| Tatoeba | 655 |
| Books | 2,049 |
| Europarl | 1,734,643 |
| Tilde | 3,434,091 |
除了 Europarl 和 Tilde 之外的所有语料库均从 Opus 收集。Europarl 和 Tilde 语料库是通过 SoftCatalà 从原始的西班牙语-加泰罗尼亚语语料库创建的合成平行语料库。
剩余的 3,272,437 句是通过对 Opus 上可用的西班牙语-德语语料库进行随机抽样，并使用 PlanTL es-ca 模型翻译成加泰罗尼亚语创建的合成平行数据。
所有数据集都经过去重和过滤，以删除任何余弦相似度小于 0.75 的句子对。这是通过使用 LaBSE 计算的句子嵌入来完成的。过滤后的数据集随后连接形成最终的 9,530,709 句平行语料库。
源语言生产者
注释
注释过程
数据集不包含任何注释。
注释者
[N/A]
个人和敏感信息
由于该数据集部分源自可能包含爬取数据的现有数据集，并且未应用特定的匿名化处理，因此数据中可能存在个人和敏感信息。在使用数据训练模型时需要考虑这一点。
使用数据集的考虑因素
数据集的社会影响
通过提供这一资源，我们旨在促进加泰罗尼亚语在自然语言处理任务中的应用，从而提高加泰罗尼亚语的可访问性和可见性。
偏见的讨论
未对该数据集应用特定的偏见缓解策略。数据中可能存在固有偏见。
其他已知限制
该数据集包含一般领域的数据。在更具体的领域（如生物医学、法律等）中应用该数据集的效用有限。
附加信息
数据集策展人
巴塞罗那超级计算中心语言技术单元（langtech@bsc.es）。
该工作得到了加泰罗尼亚政府通过 Aina 项目 的推广和资助。
许可信息
该工作根据 Attribution-NonCommercial-ShareAlike 4.0 International 许可进行许可。
引用信息
[N/A]
贡献
[N/A]
