HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT

---
dataset_info:
features:
- name: text
dtype: string
- name: id
dtype: string
- name: url
dtype: string
- name: dataset
dtype: string
splits:
- name: train
num_examples: 62119279
license: odc-by
language:
- en
size_categories:
- 10M<n<100M
tags:
- pretraining
- smol-data
pretty_name: FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT
---

# FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT

A ~100 billion token pretraining mixture combining three high-quality English data sources in a 50-30-20 ratio.

Part of the [Smol-Data](https://huggingface.co/collections/HuggingFaceFW/smol-data) collection — tried and tested mixes for strong pretraining. Inspired by [optimal dataset mixing](https://huggingface.co/blog/codelion/optimal-dataset-mixing).

## Dataset Description

| Component | Source | Tokens |
|---|---|---|
| [FinePDFs 100BT](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_100BT) | [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs) | ~50B |
| [DCLM 100BT](https://huggingface.co/datasets/HuggingFaceFW/dclm_100BT) | [DCLM-Baseline 1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) | ~30B |
| [FineWeb-Edu 100BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb_edu_100BT) | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | ~20B |

The schema is reduced to the intersection of columns across all three sources: `text`, `id`, `url`, and `dataset`.

A pre-shuffled version is available at [HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled).

## How It Was Created

The dataset was generated using [datatrove](https://github.com/huggingface/datatrove) with the [smol_data.py](https://github.com/huggingface/datatrove/blob/main/examples/smol_data.py) script. Each component was subsampled from its respective 100BT subset at the target fraction (0.5, 0.3, 0.2) using a `SamplerFilter` with seed 42. Components were written sequentially via Slurm job dependencies to avoid concurrent commits.

## Usage

```python
from datasets import load_dataset

ds = load_dataset("HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT", split="train", streaming=True)
for sample in ds:
print(sample["text"][:200])
break
```

## Citation

```bibtex
@misc{niklaus2026smoldata,
title={SmolData},
author={Joel Niklaus and Hynek Kydl{\'\i}{\v{c}}ek},
year={2026},
publisher={Hugging Face},
journal={Hugging Face repository},
howpublished={\url{https://huggingface.co/collections/HuggingFaceFW/smol-data}}
}
```