MaLA-LM/mala-monolingual-filter

MaLA Corpus（大规模语言适应语料库）是一个全面的多语言数据集，旨在支持大规模语言模型的持续预训练。它涵盖了939种语言，包含超过740亿个标记，是同类数据集中最大的之一。该数据集特别关注低资源语言的表示改进，是推进多语言模型（尤其是针对服务不足语言）的关键资源。数据集经过清洗和去重，以确保高质量的训练数据。

The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of low-resource languages, the MaLA Corpus is a critical resource for advancing multilingual models, particularly those aimed at serving underrepresented languages. The corpus is cleaned and deduplicated to ensure high-quality training data.