NLPC-UOM/nllb-top25k-ensi-cleaned

该数据集是一个用于机器翻译任务的平行语料库，包含英语（en）和僧伽罗语（si）两种语言。数据集的大小在10K到100K之间。该数据集是通过网络挖掘得到的，并且对低资源语言的语料质量进行了详细分析。研究表明，网络挖掘的语料库在不同部分之间存在显著的质量差异，并且质量因语言和数据集而异。对于某些网络挖掘的数据集，使用其最高排名的25k部分训练的神经机器翻译（NMT）模型可以与人工整理的数据集相媲美。

The dataset is used for translation tasks, including English and Sinhalese, with a size between 10K and 100K. The quality and utility of the dataset have been thoroughly studied, suitable for training Neural Machine Translation models for low-resource languages.

原始信息汇总

数据集概述

• 许可证信息: 该数据集遵循ODC-BY许可协议。
• 任务类别: 翻译
• 语言: 英语（en）和僧伽罗语（si）
• 数据规模: 10K<n<100K

引用信息

@inproceedings{ranathunga-etal-2024-quality, title = "Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora", author = "Ranathunga, Surangika and De Silva, Nisansa and Menan, Velayuthan and Fernando, Aloka and Rathnayake, Charitha", editor = "Graham, Yvette and Purver, Matthew", booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)", month = mar, year = "2024", address = "St. Julian{}s, Malta", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.eacl-long.52", pages = "860--880", abstract = "We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a similarity measure and carried out an intrinsic and extrinsic evaluation on different portions of this ranked corpus. We show that there are significant quality differences between different portions of web-mined corpora and that the quality varies across languages and datasets. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.", }