AksaraLLM/aksara-bahasa-daerah-v1

AksaraLLM Bahasa Daerah v1 是一个预训练语料库，包含印度尼西亚8种地区语言的文本数据，这些数据是从2023年11月的维基百科快照中收集的。覆盖的语言包括爪哇语、巽他语、米南加保语、亚齐语、布吉语、巴厘语、班贾尔语和马都拉语。数据集提供了每种语言的ISO代码、文章数量和相关注释。创建该数据集的动机是取代之前的数据集，提供更大量和多样化的语料库。需要注意的是，米南加保语维基百科包含许多由机器人生成的存根文章，建议在训练时进行过滤。数据集的结构包括文本和来源字段，使用CC-BY-SA 4.0许可证。

AksaraLLM Bahasa Daerah v1 is a pretraining corpus for 8 regional languages of Indonesia, collected from Wikipedia snapshot November 2023. The languages covered include Javanese, Sundanese, Minangkabau, Acehnese, Buginese, Balinese, Banjarese, and Madurese. The dataset provides ISO codes, the number of articles, and notes for each language. The motivation for creating this dataset is to replace a previous one with a more substantial and diverse corpus. Caveats are noted regarding the quality of some data, particularly for the Minangkabau language, with recommendations for filtering during training. The dataset schema includes text and source fields, and it is licensed under CC-BY-SA 4.0.