AksaraLLM/aksara-pretrain-clean-v1.1

AksaraLLM Pretrain Clean v1.1是一个用于文本生成任务的印尼语数据集。该数据集在v1版本的基础上增加了MinHash近重复数据删除步骤，减少了训练集和验证集的数量。数据集的处理方法包括使用MinHash、5-gram词片段、LSH阈值等，并继承了v1版本的其他处理步骤，如精确去重、Gopher过滤器、GlotLID语言过滤器等。数据集的许可证为CC-BY-SA 4.0。

AksaraLLM Pretrain Clean v1.1 is an Indonesian language dataset for text generation tasks. The dataset adds a MinHash near-dedup pass on top of v1, reducing the number of training and validation sets. The processing methods include MinHash, 5-gram word shingles, LSH threshold, etc., and inherit other processing steps from v1, such as exact dedup, Gopher filter, GlotLID language filter, etc. The dataset is licensed under CC-BY-SA 4.0.