riturajj-cerebras/CPT_data-30BT

# High-Quality 30B Continual Pre-Training Mixture This dataset is a carefully curated collection of approximately 30 billion high-quality tokens, specifically designed for the continual pre-training of large language models. ### ⚙️ Dataset Details * **Tokenizer:** `openai/gpt-oss-20b` * **Max Sequence Length:** 8,192 tokens * **Packing:** True ### 📊 Dataset Composition The data is blended from a diverse set of high-quality sources, heavily weighted towards code and structural knowledge to enhance reasoning and domain-specific capabilities. | Source | Weight | Tokens | |:---|---:|---:| | **dclm-edu** | 8% | 2,400,567,296 | | **finemath** | 8% | 2,457,083,904 | | **finepdfs-edu** | 8% | 2,391,719,936 | | **the-stack-v2-train-smol-ids_stars_gt10** | 35% | 10,456,367,104 | | **fineweb-edu-100BT** | 8% | 2,398,871,552 | | **fineweb2-hq** | 8% | 2,376,278,016 | | **ClimbMix** | 10% | 3,004,260,352 | | **FineWiki** | 15% | 4,518,141,952 | | **Total** | **100%** | **30,003,290,112** |# 高质量300亿Token持续预训练混合数据集 本数据集为经精心遴选的约300亿Token (Token)集合，专为大语言模型（Large Language Model）的持续预训练任务打造。 ## ⚙️ 数据集详情 * **分词器（Tokenizer）：** `openai/gpt-oss-20b` * **最大序列长度：** 8192个Token (Token) * **序列打包：** 启用 ## 📊 数据集构成 本数据集由多种高质量数据源混合而成，权重向代码与结构化知识倾斜，以强化模型的推理能力与领域专用能力。 | 数据来源 | 权重占比 | Token数量 | |:---|---:|---:| | **dclm-edu** | 8% | 2,400,567,296 | | **finemath** | 8% | 2,457,083,904 | | **finepdfs-edu** | 8% | 2,391,719,936 | | **the-stack-v2-train-smol-ids_stars_gt10** | 35% | 10,456,367,104 | | **fineweb-edu-100BT** | 8% | 2,398,871,552 | | **fineweb2-hq** | 8% | 2,376,278,016 | | **ClimbMix** | 10% | 3,004,260,352 | | **FineWiki** | 15% | 4,518,141,952 | | **总计** | **100%** | **30,003,290,112** |