reasoning-degeneration-dev/prepretraining-eval-round4-v1

---
license: mit
tags:
- prepretraining
- curriculum-learning
- data-ordering
- scaling-laws
- eval-results
---

# prepretraining-eval-round4-v1

Round 4 (150M, 5B tokens): 4 conditions, seed=42, pre-SFT and post-SFT eval PPL. Front-load overtakes constant-mix on web PPL at this scale.

## Dataset Info

- **Rows**: 8
- **Columns**: 13

## Columns

| Column | Type | Description |
|--------|------|-------------|
| run_name | Value('string') | Unique run identifier: ppt_{scale}_{condition}_d{delta}_s{seed} |
| condition | Value('string') | Data scheduling condition: baseline, front-load, constant-mix, or anneal |
| scale | Value('string') | Model scale: tiny (60M), small (150M), or medium (300M) |
| seed | Value('int64') | Random seed for reproducibility |
| is_sft | Value('bool') | Whether this run is SFT (True) or pretraining (False) |
| gold_fraction | Value('float64') | Fraction of total tokens that are gold data (delta), typically 0.05 |
| final_web_ppl | Value('float64') | Final held-out web perplexity (mean across eval files) |
| final_gold_ppl | Value('float64') | Final held-out gold perplexity (mean across 5 gold sources) |
| final_train_ppl | Value('float64') | Last logged training perplexity |
| total_steps | Value('int64') | Total training steps configured for this run |
| total_tokens_latest | Value('int64') | Latest total tokens processed (from throughput counter) |
| n_eval_web_points | Value('int64') | Number of web eval checkpoints logged |
| n_eval_gold_points | Value('int64') | Number of gold eval checkpoints logged |

## Generation Parameters

```json
{
"script_name": "analysis/extract_metrics.py + analysis/upload_results.py",
"model": "OLMo (60M/150M/300M custom configs)",
"description": "Round 4 (150M, 5B tokens): 4 conditions, seed=42, pre-SFT and post-SFT eval PPL. Front-load overtakes constant-mix on web PPL at this scale.",
"hyperparameters": {
"gold_fraction": 0.05,
"conditions": [
"baseline",
"front-load",
"constant-mix",
"anneal"
],
"lr_schedule": "cosine_with_warmup",
"optimizer": "adamw"
},
"input_datasets": [
"reasoning-degeneration-dev/prepretraining-gold-v1",
"reasoning-degeneration-dev/prepretraining-web-v1",
"reasoning-degeneration-dev/prepretraining-sft-v1"
]
}
```

## Experiment Documentation

For complete experiment details, see [https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining](https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining)

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("reasoning-degeneration-dev/prepretraining-eval-round4-v1", split="train")
print(f"Loaded {len(dataset)} rows")
```

---

*This dataset is tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)*