100B-pretokenized-mix-32k

SLM预分词100B混合数据集（32k词汇表）是一个大规模、多领域的文本数据集，专门为语言模型预训练而设计。该数据集包含约1000亿个经过预处理的token，覆盖教育内容、网络文本、百科全书资料和编程代码等多个领域。所有原始文本均使用字节对编码（BPE）分词器进行处理，该分词器拥有32,768的词汇表大小。处理后的数据以uint16整数形式存储，每个token占用2字节，并保存为1 GiB大小的二进制块文件，便于高效加载和流式处理。数据集由六个核心领域混合而成：1）Finewebedu，源自高质量教育网络内容，经过评分过滤（score ≥ 3.5）；2）Dclmedu，同样为教育网络内容，使用教育评分过滤（edu_score ≥ 3.25）；3）Cosmopedia，包含合成生成的教育性文本，涵盖故事、教程和学术内容；4）Math，专注于数学领域，使用finemath-4plus子集；5）Python，包含Python编程代码，经过ASCII字符白名单过滤以确保代码安全性；6）Wiki，源自英文维基百科的百科全书内容。各领域数据均以文档为单位进行分词，每个文档以开始标记<s>开头，以结束标记</s>结尾，所有文档连接成一个连续的token流，无内部填充，允许在训练时灵活地按任意序列长度进行重塑。该数据集适用于大规模语言模型的预训练和继续预训练任务，特别适合需要高质量、多领域文本数据的场景。其预分词格式和二进制存储方式显著减少了数据加载和预处理的开销，支持从512到8192等多种序列长度的直接重塑，便于适配不同模型架构和训练配置。数据集还提供了完整的元数据和完整性校验机制，确保数据的一致性和可靠性。The SLM Pre-tokenized 100B Mixed Dataset (32k vocabulary) is a large-scale, multi-domain text dataset specifically designed for language model pre-training. It contains approximately 100 billion preprocessed tokens, covering multiple domains such as educational content, web text, encyclopedia materials, and programming code. All original texts are processed using a Byte Pair Encoding (BPE) tokenizer with a vocabulary size of 32,768. The processed data is stored as uint16 integers, with each token occupying 2 bytes, and saved as 1 GiB binary chunk files for efficient loading and streaming. The dataset is a mixture of six core domains: 1) Finewebedu, derived from high-quality educational web content, filtered by score (score ≥ 3.5); 2) Dclmedu, also educational web content, filtered by educational score (edu_score ≥ 3.25); 3) Cosmopedia, containing synthetically generated educational texts, including stories, tutorials, and academic content; 4) Math, focused on the mathematical domain, using the finemath-4plus subset; 5) Python, containing Python programming code, filtered through an ASCII character whitelist to ensure code safety; 6) Wiki, derived from English Wikipedia encyclopedia content. Data from each domain is tokenized at the document level, with each document starting with a begin token <s> and ending with an end token </s>, and all documents are concatenated into a continuous token stream without internal padding, allowing flexible reshaping to arbitrary sequence lengths during training. The dataset is suitable for large-scale language model pre-training and continued pre-training tasks, particularly for scenarios requiring high-quality, multi-domain text data. Its pre-tokenized format and binary storage significantly reduce data loading and preprocessing overhead, supporting direct reshaping to various sequence lengths from 512 to 8192, facilitating adaptation to different model architectures and training configurations. The dataset also provides complete metadata and integrity verification mechanisms to ensure consistency and reliability.

原始信息汇总

SLM Pre-tokenized 100B Mix (32k Vocab) 数据集详情

数据集概述

该数据集是一个预分词的多领域混合语料库，总计约1000亿（100B）个token，涵盖教育、网络、百科和编程内容。所有文本均使用BPE分词器（词表大小32,768）进行分词处理，并以uint16整数格式存储为1 GiB的二进制块文件。

• 许可证: Apache-2.0
• 任务类型: 文本生成
• 语言: 英语 (en)
• 数据量: 100B < n < 1T 个tokens

数据集组成与统计

领域	来源数据集	目标配置	完成块数	处理总Token数	过滤器/设置
Finewebedu	HuggingFaceFW/fineweb-edu	30.00B	56	30.06B	score >= 3.5
Dclmedu	HuggingFaceTB/dclm-edu	30.00B	56	30.06B	edu_score >= 3.25
Cosmopedia	HuggingFaceTB/cosmopedia	11.20B	21	11.27B	无分数过滤
Math	HuggingFaceTB/finemath	9.60B	18	9.66B	finemath-4plus子集
Python	rijuludar/stackedu-en	10.20B	19	10.20B	ASCII白名单安全过滤
Wiki	HuggingFaceFW/finewiki	8.60B	16	8.59B	data/enwiki子集
总计	—	99.60B	186	99.85B	—

技术规格

• 数据类型: uint16（每个token占用2字节）
• 分词器: rijuludar/slm-tokenizer-32k（BPE，32k词表）
• 特殊token映射:
  • BOS Token（开始）: <s>
  • EOS Token（结束）: </s>
• 格式: 文档顺序格式化为 [BOS] token_1 token_2 … token_n [EOS]，所有文档连接成单一扁平uint16流，无内部填充。
• 块大小: 每个块1 GiB（1,073,741,824字节），包含536,870,912个token（2^29），可被多种序列长度整除，无需余数即可直接重塑。例如：
  • 512 序列长度 → 1,048,576 序列
  • 1024 序列长度 → 524,288 序列
  • 2048 序列长度 → 262,144 序列
  • 4096 序列长度 → 131,072 序列
  • 8192 序列长度 → 65,536 序列

各领域详细说明

1. Finewebedu

• 来源数据集: HuggingFaceFW/fineweb-edu
• 子集: sample/350BT
• 排除前缀: ["data/sample/"]
• 文本列: text
• 过滤条件: score >= 3.5（列名：score）
• 领域标签: ["fineweb-edu", "educational", "web"]

2. Dclmedu

• 来源数据集: HuggingFaceTB/dclm-edu
• 子集: 无
• 文本列: text
• 过滤条件: edu_score >= 3.25（列名：edu_score）
• 领域标签: ["dclm", "educational", "web"]

3. Cosmopedia

• 来源数据集: HuggingFaceTB/cosmopedia
• 已处理子目录:
  • data/web_samples_v2
  • data/stories
  • data/wikihow
  • data/stanford
• 文本列: text
• 过滤条件: 无
• 领域标签: ["cosmopedia", "synthetic", "educational"]

4. Math

• 来源数据集: HuggingFaceTB/finemath
• 子集: finemath-4plus
• 文本列: text
• 过滤条件: 无
• 领域标签: ["math", "finemath", "stem"]

5. Python

• 来源数据集: rijuludar/stackedu-en
• 子集: data/python
• 文本列: text
• 过滤条件: python_whitelist（可打印ASCII字符白名单，包括字母、数字、空格、制表符、换行符、回车符及常见标点符号）
• 领域标签: ["python", "code", "programming"]

6. Wiki

• 来源数据集: HuggingFaceFW/finewiki
• 子集: data/enwiki
• 文本列: text
• 过滤条件: 无
• 领域标签: ["wikipedia", "encyclopedic"]

使用方法

加载Token块

使用NumPy直接读取二进制文件并重塑为所需序列长度： python import numpy as np tokens = np.fromfile("data/finewebedu/finewebedu-0001.bin", dtype=np.uint16) seqs = tokens.reshape(-1, 2048) # 序列长度2048

完整性验证

通过SHA256哈希值与各领域目录中sample_info.json文件内的元数据进行比对，验证下载的块文件是否损坏。

目录结构

├── README.md ├── data/ │ ├── cosmopedia/cosmopedia-NNNN.bin │ ├── dclmedu/dclm-NNNN.bin │ ├── finewebedu/finewebedu-NNNN.bin │ ├── math/math-NNNN.bin │ ├── python/python-NNNN.bin │ └── wiki/wiki-NNNN.bin ├── cosmopedia/ ← 处理元数据（checkpoint.json, sample_info.json, dataset_info.json等） ├── dclmedu/ ← 处理元数据 ├── finewebedu/ ← 处理元数据 ├── math/ ← 处理元数据 ├── python/ ← 处理元数据 └── wiki/ ← 处理元数据

搜集汇总

数据集介绍

构建方式

该数据集名为100B-pretokenized-mix-32k，是一个经过预标记化的多领域语料混合集，总规模约1000亿词元。构建过程始于从六大公开数据源中筛选高质量文本：包括Fineweb-edu、DCLM-edu、Cosmopedia、Finemath、Stackedu-en以及Finewiki，分别对应教育、百科、数学、代码和百科类内容。每个领域的文本经过特定质量过滤，如教育评分阈值或ASCII白名单校验，确保数据纯净。随后，所有文本被统一送入基于字节对编码的标记器slm-tokenizer-32k（词汇表大小32,768）进行预标记化处理，生成uint16格式的词元序列。这些序列按文档边界用起始和结束标记分隔，并拼接为平坦流，再切分为1 GiB大小的二进制块文件存储，每块包含536,870,912个词元，便于后续灵活读取。

特点

该数据集的核心特点在于其高度结构化的组织形式与灵活的可用性。六大领域按照预设比例精准混合，总计186个二进制块，覆盖教育、百科、数学、代码等关键知识领域。每个块文件采用无损的原始二进制格式，无序列长度约束，训练时可根据需要自由重塑为任意长度的序列，如512、1024或8192词元，因块内词元数恰好是2的幂倍数，避免了填充或截断问题。此外，每个领域目录附带丰富的元数据文件，如sha256哈希校验信息，支持数据完整性验证。标记化过程使用固定的32k词表，并保留特殊边界标记，便于文档级采样和下游任务适配。

使用方法

使用该数据集时，用户可直接通过NumPy库加载二进制词元块文件，例如使用np.fromfile读取uint16数组，然后根据目标序列长度进行重塑，适用于语言模型预训练或微调。每个领域的数据单独存放于data/下的子目录中，文件名按序号区分。为确保数据完整性，建议利用每个领域内sample_info.json文件中存储的sha256哈希值进行校验，通过比对文件哈希来验证块未损坏。此外，元数据目录还包含检查点和数据集信息，可辅助追踪处理进度。整体流程简化为加载、校验和重塑三步，无需复杂解码步骤，直接嵌入训练管道。

背景与挑战

背景概述

在大型语言模型（LLM）预训练数据工程领域，多源异构数据的统一表征与高效处理始终是核心瓶颈。由研究者Rijul Udar主导构建的100B-pretokenized-mix-32k数据集于2025年发布，旨在为小参数规模语言模型（SLM）提供高质量、多领域的预训练语料。该数据集汇聚了来自FineWeb-Edu、DCLM-Edu、Cosmopedia、FineMath、StackEdu-EN和FineWiki六大来源的约1000亿个token，覆盖教育文本、百科全书、数学推理、编程代码与合成内容，显著提升了数据多样性与领域平衡性。通过统一的Byte-Pair Encoding（BPE）词表（32k词汇量）进行预分词，并以无序列长度约束的原始uint16数组格式存储，该数据集为下游研究者提供了灵活切片与高效加载的便利，对小型模型训练的数据标准化与规模化复现产生了重要推动作用。

当前挑战

该数据集所解决的领域问题在于，现有预训练语料多存在领域分布不均衡、词表不统一及序列长度僵化等缺陷，阻碍了小模型对多模态知识的泛化学习。通过精确分配各域token占比（教育30%、合成11.2%、数学9.6%、编程10.2%等）并采用统一分词器，该数据集缓解了跨域知识迁移与计算效率之间的冲突。构建过程中挑战显著：其一，需对来自不同来源的异构文本（如教育网页、数学论文、Python代码）实施差异化的质量过滤策略（如教育分数阈值≥3.5、ASCII安全集验证），确保语料纯净度；其二，需在内存约束下完成超100B token的流式分词与1 GiB定长分块存储，并维护每个分块的SHA256哈希校验元数据，以保障下载与加载过程中的数据完整性。

常用场景

经典使用场景

在自然语言处理与语言模型预训练的研究领域中，大规模、高质量且多样化的文本语料是构建强大语言模型的基石。100B-pretokenized-mix-32k数据集汇聚了来自教育网页、百科全书、数学文本、编程代码以及合成教育内容等六大领域的约1000亿词元，所有文本均经过统一的BPE分词器处理并以紧凑的uint16格式存储。该数据集最经典的用途在于为中小型语言模型（SLM）提供开箱即用的预训练语料，研究者可直接加载分片文件、按任意序列长度重塑，无需额外处理原始文本，极大简化了数据流水线构建流程，使得训练效率与实验可复现性显著提升。

解决学术问题

长期以来，语言模型预训练面临数据来源碎片化、质量参差不齐、分词标准不一等挑战，导致研究成果难以横向对比与复现。该数据集通过系统整合并统一处理六个领域的高质量子集，为解决上述难题提供了标准化解决方案。它有效支撑了如下学术问题：多领域知识混合对模型泛化能力的影响评估、教育性与非教育性数据配比的优化研究、以及不同分词策略下预训练效率的对比分析。其意义在于为社区提供了一个可复现的基准语料，使得后续关于模型架构、训练算法及数据配比的研究能够基于一致的数据基础展开，推动了小规模语言模型研究的规范化与可比较性。

衍生相关工作

围绕100B-pretokenized-mix-32k数据集，已衍生出若干经典研究方向与后续工作。最直接的相关工作包括基于该语料训练并发布的一系列小型语言模型（SLM），这些模型在参数量低于1B的情况下，在常识推理、数学计算与代码生成等基准上取得了可观成绩。此外，研究者利用该数据集的领域标签探索了课程学习（curriculum learning）策略，通过调整不同子集的引入顺序来加速收敛。另一重要方向是数据配比实验，社区基于该语料系统地分析了教育网页、百科、代码等各领域数据的占比对模型下游性能的影响，为构建更高效的语言模型预训练混合数据提供了实证指导。这些工作共同构成了围绕标准化预分词语料进行高效模型训练与方法验证的完整生态。