jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining

mmBERT预训练数据集P2是用于训练mmBERT模型套件的多样化多语言预训练数据混合体（训练了2.3万亿个标记）。该数据集包含多种数据源，如高质量的多语言网络爬取数据、英文网络爬取数据、代码库和文件、学术预印本、问答论坛、指令跟随数据、数学内容、科学论文、百科全书文章以及文学和参考书籍等。数据集以MDS格式提供，适用于Composer和ModernBERT训练库。该阶段涵盖了60种语言以及代码，采用逆温度抽样计划。

mmBERT Pre-training Data P2 is a diverse multilingual pre-training data mixture used to train the mmBERT model suite (trained for 2.3T tokens). The dataset includes various data sources such as high-quality multilingual web crawl data, English web crawl data, code repositories and files, academic preprints, Q&A forums, instruction-following data, mathematical content, scientific papers, encyclopedia articles, and literature and reference books. The dataset is provided in MDS format, suitable for use with Composer and ModernBERT training library. This phase covers 60 languages plus code, using an inverse temperature sampling schedule.