riturajj-cerebras/CPT_data-30BT

# High-Quality 30B Continual Pre-Training Mixture

This dataset is a carefully curated collection of approximately 30 billion high-quality tokens, specifically designed for the continual pre-training of large language models.

### ⚙️ Dataset Details
* **Tokenizer:** `openai/gpt-oss-20b`
* **Max Sequence Length:** 8,192 tokens
* **Packing:** True

### 📊 Dataset Composition

The data is blended from a diverse set of high-quality sources, heavily weighted towards code and structural knowledge to enhance reasoning and domain-specific capabilities.

| Source | Weight | Tokens |
|:---|---:|---:|
| **dclm-edu** | 8% | 2,400,567,296 |
| **finemath** | 8% | 2,457,083,904 |
| **finepdfs-edu** | 8% | 2,391,719,936 |
| **the-stack-v2-train-smol-ids_stars_gt10** | 35% | 10,456,367,104 |
| **fineweb-edu-100BT** | 8% | 2,398,871,552 |
| **fineweb2-hq** | 8% | 2,376,278,016 |
| **ClimbMix** | 10% | 3,004,260,352 |
| **FineWiki** | 15% | 4,518,141,952 |
| **Total** | **100%** | **30,003,290,112** |