MaLA-LM/mala-bilingual-translation-corpus

MaLA语料库是一个旨在增强大规模语言适应性的综合多语种数据集，包含2500种以上语言对的双语翻译数据。该数据集通过从不同来源收集数据并进行严格的预处理（包括清洗、去重和标准化）来确保数据质量。适用于大型语言模型的持续预训练、多语言基准测试的模型微调以及多语言任务，如机器翻译。

The MaLA Corpus is a comprehensive multilingual dataset designed to enhance massive language adaptation, containing bilingual translation data for over 2,500 language pairs. The dataset is created by collecting data from various sources and rigorously pre-processing it (including cleaning, deduplication, and normalization) to ensure data quality. It is suitable for continual pre-training of large language models, fine-tuning models on multilingual benchmarks, and multilingual tasks such as machine translation.