SEACrowd/korpus_nusantara

Korpus Nusantara数据集是一个并行语料库，收集自Tanjungpura大学信息学专业学生的研究、作业和论文。该数据集主要用于机器翻译任务，涵盖了印尼语到25种印尼地方方言的翻译。由于并非所有方言都有ISO639-3标准编码，数据集根据最接近的语言家族进行了分组，包括爪哇语、达雅克语、布吉语、巽他语、马都拉语、班贾尔语、巴塔克托巴语、客家话、马来语、米南加保语和潮州话。数据集可以自由用于研究目的，但需要引用相关论文。

The Korpus Nusantara dataset is a parallel corpus collected from studies, assignments, and theses of students in the Informatics Study Program at Tanjungpura University. This dataset is primarily used for machine translation tasks, covering translations from Indonesian to 25 local dialects in Indonesia. Since not all dialects have ISO639-3 standard coding, the dataset is grouped into the closest language families, including Javanese, Dayak, Buginese, Sundanese, Madurese, Banjar, Batak Toba, Khek, Malay, Minangkabau, and Tiociu. The dataset can be freely used for research purposes, provided that the relevant papers are cited.

原始信息汇总

数据集概述

语言

• ind
• jav
• xdy
• bug
• sun
• mad
• bjn
• bbc
• khek
• msa
• min
• tiociu

支持的任务

• 机器翻译

数据集使用

使用 datasets 库

python from datasets import load_dataset dset = datasets.load_dataset("SEACrowd/korpus_nusantara", trust_remote_code=True)

使用 seacrowd 库

python import seacrowd as sc

加载数据集使用默认配置

dset = sc.load_dataset("korpus_nusantara", schema="seacrowd")

检查数据集的所有可用子集（配置名称）

print(sc.available_config_names("korpus_nusantara"))

使用特定配置加载数据集

dset = sc.load_dataset_by_config_name(config_name="<config_name>")

数据集主页

• https://github.com/herrysujaini/korpusnusantara

数据集版本

• 源版本: 1.0.0
• SEACrowd版本: 2024.06.20

数据集许可证

• 未知

引用

如果使用 Korpus Nusantara 数据集，请引用以下内容： plaintext @article{sujaini2020improving, title={Improving the role of language model in statistical machine translation (Indonesian-Javanese)}, author={Sujaini, Herry}, journal={International Journal of Electrical and Computer Engineering}, volume={10}, number={2}, pages={2102}, year={2020}, publisher={IAES Institute of Advanced Engineering and Science} }

@article{lovenia2024seacrowd, title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages}, author={Holy Lovenia and Rahmad Mahendra and Salsabil Maulana Akbar and Lester James V. Miranda and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Joseph Marvin Imperial and Onno P. Kampman and Joel Ruben Antony Moniz and Muhammad Ravi Shulthan Habibi and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Joanito Agili Lopo and William Nixon and Börje F. Karlsson and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Jan Christian Blaise Cruz and Chenxi Whitehouse and Ivan Halim Parmonangan and Maria Khelli and Wenyu Zhang and Lucky Susanto and Reynard Adha Ryanda and Sonny Lazuardi Hermawan and Dan John Velasco and Muhammad Dehan Al Kautsar and Willy Fitra Hendria and Yasmin Moslem and Noah Flynn and Muhammad Farid Adilazuarda and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Muhammad Reza Qorib and Amirbek Djanibekov and Wei Qi Leong and Quyet V. Do and Niklas Muennighoff and Tanrada Pansuwan and Ilham Firdausi Putra and Yan Xu and Ngee Chia Tai and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Alham Fikri Aji and Sedrick Keh and Genta Indra Winata and Ruochen Zhang and Fajri Koto and Zheng-Xin Yong and Samuel Cahyawijaya}, year={2024}, eprint={2406.10118}, journal={arXiv preprint arXiv: 2406.10118} }