SEACrowd/minangnlp_mt

Minangnlp Mt数据集是一个米南加保语-印尼语（MIN-ID）的平行语料库，通过维基百科创建。该数据集包含16,371对高质量的平行句子，适用于机器翻译任务。数据集的创建过程包括从维基百科获取文章、通过标题匹配对齐文档、进行句子分割、使用双语词典翻译、通过ROUGE-1评分进行句子对齐，并最终通过人工评估确保数据质量。数据集的质量评估结果显示，其充分性和流畅性评分分别为4.98和4.87，表明该语料库非常适合用于机器翻译训练。

The Minangnlp Mt dataset is a Minangkabau–Indonesian (MIN-ID) parallel corpus created using Wikipedia. It contains 16,371 high-quality parallel sentence pairs suitable for machine translation tasks. The dataset creation process involves obtaining articles from Wikipedia, aligning documents through title matching, performing sentence segmentation, translating using a bilingual dictionary, aligning sentences using ROUGE-1 scores, and finally conducting manual evaluation to ensure data quality. The quality assessment results show adequacy and fluency scores of 4.98 and 4.87, respectively, indicating that the corpus is highly suitable for machine translation training.

原始信息汇总

数据集概述

该数据集是一个用于机器翻译的Minangkabau-Indonesian（MIN-ID）平行语料库，通过维基百科创建。数据集包含224,180篇Minangkabau文章和510,258篇Indonesian文章，通过标题匹配对齐文档，得到111,430对MIN-ID文档。经过句子分割和使用双语词典翻译，最终得到345,146对MIN-ID平行句子。通过进一步过滤和人工评估，最终得到16,371对高质量的MIN-ID平行句子。

语言

• Minangkabau (min)
• Indonesian (ind)

支持的任务

• 机器翻译

数据集使用

使用 datasets 库

python from datasets import load_dataset dset = datasets.load_dataset("SEACrowd/minangnlp_mt", trust_remote_code=True)

使用 seacrowd 库

python import seacrowd as sc

使用默认配置加载数据集

dset = sc.load_dataset("minangnlp_mt", schema="seacrowd")

查看数据集的所有可用子集（配置名称）

print(sc.available_config_names("minangnlp_mt"))

使用特定配置加载数据集

dset = sc.load_dataset_by_config_name(config_name="<config_name>")

数据集版本

• 源版本: 1.0.0
• SEACrowd版本: 2024.06.20

数据集许可证

MIT

引用

bibtex @inproceedings{koto-koto-2020-towards, title = "Towards Computational Linguistics in {M}inangkabau Language: Studies on Sentiment Analysis and Machine Translation", author = "Koto, Fajri and Koto, Ikhwan", booktitle = "Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation", month = oct, year = "2020", address = "Hanoi, Vietnam", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.paclic-1.17", pages = "138--148", }

@article{lovenia2024seacrowd, title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages}, author={Holy Lovenia and Rahmad Mahendra and Salsabil Maulana Akbar and Lester James V. Miranda and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Joseph Marvin Imperial and Onno P. Kampman and Joel Ruben Antony Moniz and Muhammad Ravi Shulthan Habibi and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Joanito Agili Lopo and William Nixon and Börje F. Karlsson and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Jan Christian Blaise Cruz and Chenxi Whitehouse and Ivan Halim Parmonangan and Maria Khelli and Wenyu Zhang and Lucky Susanto and Reynard Adha Ryanda and Sonny Lazuardi Hermawan and Dan John Velasco and Muhammad Dehan Al Kautsar and Willy Fitra Hendria and Yasmin Moslem and Noah Flynn and Muhammad Farid Adilazuarda and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Muhammad Reza Qorib and Amirbek Djanibekov and Wei Qi Leong and Quyet V. Do and Niklas Muennighoff and Tanrada Pansuwan and Ilham Firdausi Putra and Yan Xu and Ngee Chia Tai and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Alham Fikri Aji and Sedrick Keh and Genta Indra Winata and Ruochen Zhang and Fajri Koto and Zheng-Xin Yong and Samuel Cahyawijaya}, year={2024}, eprint={2406.10118}, journal={arXiv preprint arXiv: 2406.10118} }