MaLA-LM/mala-monolingual-split
收藏Hugging Face2025-10-22 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/MaLA-LM/mala-monolingual-split
下载链接
链接失效反馈官方服务：
资源简介：
MaLA Corpus（大规模语言适应语料库）是一个全面的多语言数据集，旨在支持大规模语言模型的持续预训练。它涵盖了939种语言，包含超过740亿个标记，是同类数据集中最大的之一。该数据集特别关注低资源语言的表示改进，是推动多语言模型发展的重要资源，尤其是那些旨在服务于代表性不足语言的模型。数据集经过清洗、去重和标准化处理，以确保数据质量。
The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of low-resource languages, the MaLA Corpus is a critical resource for advancing multilingual models, particularly those aimed at serving underrepresented languages.
提供机构：
MaLA-LM
