riturajj-cerebras/CPT_data-100BT
收藏Hugging Face2026-04-09 更新2026-04-12 收录
下载链接：
https://hf-mirror.com/datasets/riturajj-cerebras/CPT_data-100BT
下载链接
链接失效反馈官方服务：
资源简介：
This dataset is a carefully curated collection of approximately 100 billion high-quality tokens, specifically designed for the continual pre-training of large language models.
### ⚙️ Dataset Details
* **Tokenizer:** `gpt-oss-20b`
* **Max Sequence Length:** 8,192 tokens
* **Packing:** True
### 📊 Dataset Composition
The data is blended from a diverse set of high-quality sources, heavily weighted towards code and structural knowledge to enhance reasoning and domain-specific capabilities. *(Note: The weights below reflect the actual proportion of selected tokens, which were dynamically adjusted during mixing to account for source-specific size constraints).*
| Source | Effective Weight | Tokens |
|:---|---:|---:|
| **dclm-edu** | 8.01% | 8,007,901,184 |
| **finemath** | 8.00% | 8,004,698,112 |
| **finepdfs-edu** | 7.99% | 7,993,802,752 |
| **the-stack-v2-train-smol-ids_stars_gt10** | 29.58% | 29,576,740,864 |
| **fineweb-edu-100BT** | 7.99% | 7,993,630,720 |
| **fineweb2-hq** | 13.43% | 13,425,991,680 |
| **ClimbMix** | 10.00% | 10,000,375,808 |
| **FineWiki** | 15.00% | 14,999,977,984 |
| **Total** | **100.00%** | **100,003,119,104** |
提供机构：
riturajj-cerebras
