ontocord/MixtureVitae-v1-decontaminated

---
License: odc-by
---

# MixtureVitae (Decontaminated)

## Dataset Summary

**MixtureVitae** is a **422B-token open pretraining dataset** introduced in the paper
[*MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources*](https://arxiv.org/abs/2509.25531).

The dataset is designed to answer a key question:
> *Can we train competitive large language models using only permissive-licensed and low-risk data, without resorting to unrestricted web scrapes?*

To this end, MixtureVitae prioritizes **permissive licensing, legal safety, and transparent provenance** while still maintaining high performance across reasoning, instruction following, and general NLP tasks.

## Dataset Composition

MixtureVitae integrates three major components (≈ 422B tokens total):

- **Curated Sources (~210B tokens)**
High-quality domain text: SEC filings, arXiv/PubMed, patents, MegaWika, science/news/legal corpora, The Stack v1 code (~12% of total).

- **Instruction & Reasoning (~178B tokens)**
Synthetic instruction/QA/math/code data, generated from permissive seeds (e.g., Magpie, MetaMathQA, OpenMathInstruct, UltraFeedback, Glaive-AI, OpenThoughts).

- **Web (~34B tokens)**
Selected permissive or re-filtered crawls (Nemotron-CC, MagaCorpus, FineFineWeb).

**By license tier:**
- Tier 1: 352B (explicit open licenses & PD)
- Tier 2: 52B (curated permissive repositories like The Stack v1)
- Tier 3: 18B (civic/government works)

## Dataset Structure

Each example in MixtureVitae consists of one or more documents concatenated into a text sequence.

- Documents are separated by the special token: `<|endoftext|>`. We recommend replacing this token with your appropriate `eos` token from the target tokenizer used for training your model.
- We have used `<think>` and `</think>` tokens in some reasoning datasets. You may wish to add these special tokens to your tokenizer.

## Decontamination**
Decontaminated against the test sets for 18 benchmarks.

| Benchmark | Category |
|-----------|----------|
| [IFEval](https://huggingface.co/datasets/google/IFEval) | Instruction following |
| [ARC](https://huggingface.co/datasets/allenai/ai2_arc) | Science QA |
| [COPA](https://huggingface.co/datasets/pkavumba/balanced-copa) | Commonsense reasoning |
| [LAMBADA](https://huggingface.co/datasets/EleutherAI/lambada_openai) | Language modeling |
| [OpenBookQA](https://huggingface.co/datasets/allenai/openbookqa) | Science QA |
| [Winogrande](https://huggingface.co/datasets/allenai/winogrande) | Commonsense reasoning |
| [BoolQ](https://huggingface.co/datasets/google/boolq) | Reading comprehension |
| [HellaSwag](https://huggingface.co/datasets/Rowan/hellaswag) | Commonsense reasoning |
| [PIQA](https://huggingface.co/datasets/ybisk/piqa) | Physical intuition QA |
| [GSM8K](https://huggingface.co/datasets/openai/gsm8k) | Math reasoning |
| [ALERT](https://huggingface.co/datasets/Babelscape/ALERT) | Safety |
| [GPQA](https://huggingface.co/datasets/Idavidrein/gpqa) | Graduate-level science |
| [MATH](https://huggingface.co/datasets/hendrycks/competition_math) | Math reasoning |
| [MBPP](https://huggingface.co/datasets/google-research-datasets/mbpp) | Code |
| [HumanEval](https://huggingface.co/datasets/openai/openai_humaneval) | Code |
| [SimpleQA](https://huggingface.co/datasets/openai/simple-evals) | Factual QA |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | Commonsense reasoning |
| [DoNotAnswer](https://huggingface.co/datasets/LibrAI/do-not-answer) | Safety |

## Limitations & Considerations

- Not 100% free of legal risk; license heuristics may miss edge cases.
- No full cross-dataset deduplication → potential near-duplicates.
- Domain balance favors reasoning/math/instruction, underrepresents other genres.

## How to Cite

```bibtex
@misc{nguyen2025mixturevitaeopenwebscalepretraining,
title={MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources},
author={Huu Nguyen and Victor May and Harsh Raj and Marianna Nezhurina and Yishan Wang and Yanqi Luo and Minh Chien Vu and Taishi Nakamura and Ken Tsui and Van Khue Nguyen and David Salinas and Aleksandra Krasnodębska and Christoph Schuhmann and Mats Leon Richter and Xuan-Son and Vu and Jenia Jitsev},
year={2025},
eprint={2509.25531},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2509.25531},
}