Puro-2B

# Puro-2B Pretraining Data: The Recipe Behind a 2B Model **This is the materialized pretraining data release for [Puro-2B-Base](https://huggingface.co/thu-pacman/Puro-2B-Base), a 2B base model trained from scratch on consumer-grade RTX 5090 GPUs.** [](https://huggingface.co/thu-pacman/Puro-2B-Base) [](https://huggingface.co/datasets/thu-pacman/Puro-2B) [](#license-and-upstream-terms)  The repository contains the component-level data pools used to construct the two Puro-2B pretraining phases, together with the tokenizer used for token accounting. It is organized for inspection, selective streaming, and recipe reconstruction rather than as a small train/test benchmark. ## Dataset Summary | Domain | Phase 1 | Phase 1 share | Phase 2 | Phase 2 share | | --- | ---: | ---: | ---: | ---: | | English | 321B | 73.2% | 558B | 59.4% | | Mathematics | 31.6B | 7.2% | 171B | 18.3% | | Chinese | 51.3B | 11.7% | 88.5B | 9.4% | | Code | 34.6B | 7.9% | 108B | 11.5% | | SFT / instruction-formatted | 0 | 0.0% | 12.7B | 1.3% | | **Materialized pool** | **439B** | **100.0%** | **938B** | **100.0%** | Token counts are computed with the bundled tokenizer and measure materialized token exposure, not unique upstream content. Dataset-family rows may aggregate overlapping configurations. The production model consumed approximately 439B tokens in Phase 1 and 961B additional tokens in Phase 2. The Phase 2 training count is larger than the 938B stationary Phase 2 pool because the opening transition replays Phase 1 data while introducing the Phase 2 stream. Replay is a training-schedule operation, not additional independent source data. <p align="center"> <img src="./assets/data_domain_composition.png" width="600px" alt="Puro-2B Phase 1 and Phase 2 domain composition"> </p> ## Repository Structure ```text . ├── phase1/ # Component data used to build the Phase 1 mixture ├── phase2/ # Component data used to build the Phase 2 mixture ├── qwen2_tokenizer/ # Tokenizer files used for token accounting └── README.md ``` The release defines two Hugging Face configurations, `phase1` and `phase2`. Each configuration exposes one `train` split because both phases are pretraining corpora. Directory names such as `train`, `validation`, `test`, or `benchmark` inside individual source components preserve upstream organization; they are not Puro-2B evaluation splits. Files are grouped by source component. Their filesystem order is not the final curriculum order used by the production run. The transition, replay, curriculum bucket construction, deterministic seeds, and global uniform reshuffle are recipe operations documented in the technical report and implemented in the data-processing pipeline. ## Data Fields Released Parquet records use a common schema: | Field | Type | Description | | --- | --- | --- | | `text` | string | Normalized text presented to the tokenizer. | | `token_count` | int64 | Number of tokens under the bundled tokenizer. | | `domain_category` | string | Semantic domain, such as English, Chinese, math, code, or instruction-formatted data. | | `source_component_id` | string | Stable identifier for the materialized source component. | Source-specific metadata not represented by these fields should be recovered from the upstream dataset and the Puro data-processing manifests. ## Loading the Data The full repository is hundreds of gigabytes. Streaming or selecting individual components is recommended for exploration. ```python from datasets import load_dataset phase1 = load_dataset( "thu-pacman/Puro-2B", "phase1", split="train", streaming=True, ) example = next(iter(phase1)) print(example["source_component_id"], example["token_count"]) print(example["text"][:500]) ``` Load Phase 2 by changing the configuration name: ```python phase2 = load_dataset( "thu-pacman/Puro-2B", "phase2", split="train", streaming=True, ) ``` For a single component, load its Parquet files directly: ```python from datasets import load_dataset open_web_math = load_dataset( "parquet", data_files={ "train": ( "hf://datasets/thu-pacman/Puro-2B/" "phase2/open-web-math/*.parquet" ) }, split="train", streaming=True, ) ``` For reproducible experiments, pin the dataset `revision` to a commit hash and record the selected component paths, filtering rules, and shuffle seed. ## How the Recipe Was Built The Puro-2B pipeline separates recipe selection from shard materialization: 1. **Source acquisition.** Candidate English, Chinese, mathematics, code, and instruction-formatted datasets are collected from public sources. 2. **Within-source processing.** Large web components are deduplicated within source. The project does not claim global cross-source deduplication. 3. **Proxy benchmarking.** A shared Qwen3-0.6B proxy checkpoint is continued on candidate sources or score slices and evaluated on a fixed 15-benchmark suite. These results guide source and within-source selection. 4. **Materialization.** Selected revisions, filters, token budgets, mixture weights, and random seeds are frozen into component-level Parquet files. 5. **Training order.** Phase 2 uses component-local ranks to build an ordered curriculum while approximately preserving the mixture. Scores are never compared numerically across different source datasets. Because the proxy study has no matched base-mixture-only continuation, its candidate profiles should be interpreted as recipe signals rather than isolated causal effects. <p align="center"> <img src="./assets/proxy_benchmark_protocol.png" width="600px" alt="Puro-2B proxy benchmarking protocol"> </p> ## Sources at a Glance | Domain | Representative source families | | --- | --- | | English | Nemotron-CC-v2, FineWeb-Edu, Cosmopedia-v2, DCLM, ArXiv, FineWiki | | Chinese | FineWeb-Edu-CN, ChineseWebText2.0, merged Chinese web corpora, Baidu Baike, FineWiki-CN, UN documents, Alpaca-Zh | | Mathematics | UltraData-Math, SwallowMath-v2, Nemotron-CC-Math, MegaMath-Web-Pro, OpenWebMath, FineMath, AutoMathText, NuminaMath-CoT, FineProofs | | Code | MegaMath-Code, Nemotron Synthetic Code, Swallow-Code-v2, CoderForge, StackExchange, Python and GitHub corpora, Codeforces-CoTs, Jupyter Agent | | Instruction-formatted | Nemotron Terminal Corpus, JiuZhang3.0, Tulu 3 SFT, ToolMind, ToolBench, SlimOrca, LongAlpaca, OpenThoughts Agent | See the [technical report source](https://github.com/thu-yao-01-luo/Prom-Technical-Report) for the component-level token counts, materialization modes, upstream identifiers, proxy protocol, and license audit. ## License and Upstream Terms The dataset card uses the **`other`** license designation because the recipe combines components with different upstream terms. Apache 2.0 applies only to project-authored documentation, metadata, manifests, and processing artifacts explicitly released under that license. It does not replace the licenses, copyrights, privacy rights, or terms attached to third-party source content. Important boundaries include: - NVIDIA-controlled sources such as Nemotron-CC-v2 and the Synthetic-Code partition of Nemotron-Pretraining-Code-v1 have data terms that prohibit raw redistribution. Their corresponding directories provide reconstruction or provenance notices instead of republished raw samples where required. - Web and code corpora may retain terms from Common Crawl, websites, authors, or original software repositories even when dataset metadata uses a permissive license. - Some source cards do not declare a dataset-level license, and some mixtures contain subsets with stricter terms. Absence of a declared license should not be interpreted as permission to redistribute or use the content without restriction. Users are responsible for checking the upstream terms of the components they download and for determining whether their intended use is permitted. The technical report's component inventory is the starting point for that review. ## Known Limitations and Responsible Use This is large-scale pretraining data collected from heterogeneous public and synthetic sources. It may contain personal information, copyrighted material, incorrect claims, duplicated content, offensive language, security-sensitive code, or social and cultural biases. Filtering reduces some problems but does not eliminate them. The release does not claim exhaustive benchmark decontamination. Some source families provide decontaminated subsets, but that property should not be generalized to every component. Users should run task-specific contamination, privacy, safety, and licensing checks before training or deploying a model. ## Related Artifacts - Model: [thu-pacman/Puro-2B-Base](https://huggingface.co/thu-pacman/Puro-2B-Base) - Training code: [thu-pacman/Puro-Megatron](https://github.com/thu-pacman/Puro-Megatron) - Data processing: [thu-pacman/Kaiyuan-Spark](https://github.com/thu-pacman/Kaiyuan-Spark) - Technical report: pending upload to arXiv. ## Citation Our technical report is pending upload to arXiv. Please come back later for an official link! Meanwhile, please use this citation: ```bibtex @misc{luo2026puro2b, title = {Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within {\$}5090}, author = {Kairong Luo and Jiarui Cui and Yaorui Yin and Shengqi Chen and Yiming Yang and Linxiang Gao and Yanmohan Wang and Chengxia Li and Mingzhe Zhang and Kaifeng Lyu and Wenguang Chen}, year = {2026}, url = {https://huggingface.co/thu-pacman/Puro-2B-Base} } ```