MaLA-LM/mala-monolingual-dedup

MaLA Corpus（大规模语言适应语料库）是一个多语言数据集，涵盖939种语言，包含超过740亿个词元。该数据集经过清洗、去重和标准化处理，旨在支持大规模语言模型的持续预训练，特别是针对低资源语言的表示改进。数据集的结构包括语言覆盖、词元数量，并明确了其预期用途，如持续预训练、多语言任务和模型微调。

The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. The dataset is focused on improving the representation of low-resource languages, making it a critical resource for advancing multilingual models, particularly those aimed at serving underrepresented languages.