kshitijthakkar/nemotron-balanced-1b-v2

# Nemotron Balanced 1B Token Dataset

## Overview
This dataset is a balanced 1 billion token subset of NVIDIA's Nemotron-Pretraining-Specialized-v1 and v1.1 datasets.

## Statistics
- **Total Samples**: 1,299,680
- **Total Tokens**: 999,999,950
- **Tokenizer**: Qwen/Qwen3-0.6B
- **Random Seed**: 42

## Subset Distribution

| Subset | Samples | Tokens | Target | Completion |
|--------|---------|--------|--------|------------|
| Nemotron-Pretraining-Math-Textbooks | 74,593 | 150,000,000 | 150,000,000 | 100.0% |
| Nemotron-Pretraining-STEM-SFT | 23,755 | 120,000,000 | 120,000,000 | 100.0% |
| Nemotron-Pretraining-Scientific-Coding | 117,174 | 149,999,966 | 150,000,000 | 100.0% |
| Nemotron-Pretraining-Wiki-Rewrite | 102,778 | 120,000,000 | 120,000,000 | 100.0% |
| Nemotron-Pretraining-RQA | 13,406 | 100,000,000 | 100,000,000 | 100.0% |
| Nemotron-Pretraining-InfiniByte-Reasoning | 6,316 | 110,000,000 | 110,000,000 | 100.0% |
| Nemotron-Pretraining-Formal-Logic | 411,082 | 60,000,000 | 60,000,000 | 100.0% |
| Nemotron-Pretraining-Economics | 295,321 | 39,999,984 | 40,000,000 | 100.0% |
| Nemotron-Pretraining-Multiple-Choice | 113,592 | 50,000,000 | 50,000,000 | 100.0% |
| Nemotron-Pretraining-Unconditional-Algorithmic | 33,069 | 50,000,000 | 50,000,000 | 100.0% |
| Nemotron-Pretraining-Code-Concepts | 108,594 | 50,000,000 | 50,000,000 | 100.0% |

## Usage

```python
from datasets import load_from_disk

# Load the dataset
dataset = load_from_disk("./data/nemotron_balanced_1b_v2/hf_dataset")

# Or load from parquet
import pandas as pd
df = pd.read_parquet("./data/nemotron_balanced_1b_v2/dataset.parquet")
```

## Dataset Fields
- **text**: The raw text content
- **formatted_text**: The formatted text (same as text for pretraining data)
- **encoded_text**: Tokenized version of the text (list of token IDs)
- **source**: Source subset name
- **token_count**: Number of tokens in the sample

## License
Follows the licensing of NVIDIA Nemotron-Pretraining-Specialized-v1 and v1.1:
- Most subsets: CC BY 4.0
- Wiki-Rewrite and Scientific-Coding (v1): CC BY-SA 4.0 and GFDL

## Citation
```bibtex
@misc{nemotron-balanced-1b,
title={Nemotron Balanced 1B Token Dataset},
author={Created from NVIDIA Nemotron-Pretraining-Specialized-v1},
year={2024},
note={Tokenized with Qwen/Qwen3-0.6B}
}
```