MathPile

MathPile是一个包含约9.5亿个标记的大规模数学中心语料库，用于预训练数学模型。在创建过程中，我们坚持‘少即是多’的原则，坚信数据质量优于数量，即使在预训练阶段也是如此。我们的数据收集和处理工作包括复杂的预处理、预过滤、语言识别、清洗、过滤和去重，确保了语料库的高质量。此外，我们还对下游基准测试集进行了数据污染检测，以消除重复项。

MathPile is a large-scale math-centric corpus containing approximately 950 million tokens, designed for pretraining mathematical models. During its creation, we adhered to the principle of "less is more", firmly believing that data quality outweighs quantity, even during the pretraining stage. Our data collection and processing workflow includes complex preprocessing, pre-filtering, language identification, cleaning, filtering, and deduplication, which ensures the high quality of the corpus. In addition, we conducted data contamination detection on downstream benchmark datasets to eliminate duplicate entries.

搜集汇总

背景与挑战

背景概述

MathPile是一个用于预训练数学模型的大规模数学中心语料库，包含约9.5亿个标记。其特点在于强调数据质量优于数量，通过复杂的预处理、过滤和去重等步骤确保高质量，并进行了下游基准测试集的数据污染检测。