Xerv-AI/PURE

PURE（Pretraining Universal Reasoning Engine）数据集是一个大规模、经过超滤和结构统一的语料库，旨在激发基础大型语言模型的链式思考（CoT）推理能力。该数据集通过聚合最强大的推理、编码、数学和对齐数据集，并将其严格标准化为三列管道（问题、推理、答案），从而弥补标准指令调优数据集的不足，教会模型“如何”思考。它去除了不可解析的二进制文件、音频转录和损坏的数据结构，为中小参数模型（例如10亿规模）提供了一个干净、CPU内存友好且高密度的课程，以学习结构化逻辑，而无需大规模计算集群进行数据清理。数据集主要用于从零开始预训练、持续预训练（CPT）和监督微调（SFT），支持文本生成、问答和摘要等任务。数据来源包括16个开源数据集，涵盖复杂推理与数学（如GPQA、MMLU-Pro）、通用指令与链式思考（如GLM-5.1-Reasoning-1M-Cleaned）、软件工程与内核优化（如SWE-bench_Pro）、人类对齐与对话流（如Anthropic/hh-rlhf）以及基础指令调优（如Alpaca-cleaned）。数据集为纯英文（主要），包含多语言STEM子集，采用混合许可证（源自各自源数据集，严格用于非商业研究目的）。局限性包括仅限文本模态、推理稀疏性以及继承自源数据集的偏见。The PURE (Pretraining Universal Reasoning Engine) dataset is a massive, hyper-filtered, and structurally unified corpus designed to ignite chain-of-thought (CoT) reasoning capabilities in foundational large language models. It bridges the gap left by standard instruction-tuning datasets by aggregating the most robust reasoning, coding, mathematical, and alignment datasets available, funneling them through a strict three-column standardization pipeline (question, reasoning, answer). By stripping away unparseable binaries, audio transcripts, and broken data structures, PURE offers a clean, CPU-memory-friendly, and highly dense curriculum for small-to-medium parameter models (e.g., 1B scale) to learn structured logic without requiring massive compute clusters for data cleaning. The dataset is primarily used for pretraining from scratch, continuous pretraining (CPT), and supervised fine-tuning (SFT), supporting tasks such as text-generation, question-answering, and summarization. It is sourced from 16 open-source datasets covering complex reasoning & mathematics (e.g., GPQA, MMLU-Pro), general instruction & chain-of-thought (e.g., GLM-5.1-Reasoning-1M-Cleaned), software engineering & kernel optimization (e.g., SWE-bench_Pro), human alignment & conversational flow (e.g., Anthropic/hh-rlhf), and base instruction tuning (e.g., Alpaca-cleaned). The dataset is predominantly in English with multi-lingual STEM subsets and uses a mixed license (derived from respective source datasets, strictly for non-commercial research purposes). Limitations include being strictly text-based, reasoning sparsity, and inherited biases from source datasets.