NLPC-UOM/nllb-top25k-enta-cleaned

该数据集是一个包含英语和泰米尔语的平行语料库，主要用于翻译任务。数据集来源于网络挖掘，研究对其质量进行了详细分析，并展示了不同部分的语料库在神经机器翻译（NMT）模型中的表现。研究表明，某些网络挖掘的数据集在训练NMT模型时，其最高排名的25k部分可以与人工整理的语料库相媲美。

This dataset is a parallel corpus containing English and Tamil, primarily used for translation tasks. The dataset is sourced from web mining, and a detailed analysis of its quality was conducted, demonstrating the performance of different portions of the corpus in Neural Machine Translation (NMT) models. The research shows that for some web-mined datasets, the highest-ranked 25k portion can be on par with human-curated datasets when training NMT models.

原始信息汇总

数据集概述

• 许可证信息: 该数据集遵循ODC-BY许可证。
• 任务类别: 翻译
• 语言: 英语 (en) 和 泰米尔语 (ta)
• 数据规模: 10K<n<100K

引用信息

@inproceedings{ranathunga-etal-2024-quality, title = "Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora", author = "Ranathunga, Surangika and De Silva, Nisansa and Menan, Velayuthan and Fernando, Aloka and Rathnayake, Charitha", editor = "Graham, Yvette and Purver, Matthew", booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)", month = mar, year = "2024", address = "St. Julian{}s, Malta", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.eacl-long.52", pages = "860--880", abstract = "We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a similarity measure and carried out an intrinsic and extrinsic evaluation on different portions of this ranked corpus. We show that there are significant quality differences between different portions of web-mined corpora and that the quality varies across languages and datasets. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.", }