albertlungu/final-nous-corpus

# Nous Training Corpus ## Dataset Description This dataset contains the pre-training corpus for the Nous multimodal model. - **Total tokens:** 170,756,572,334 (~170.8B) - **Uncompressed size:** 651.4 GB - **Tokenizer:** TikToken cl100k_base - **Format:** Plain text, double-newline separated documents ## Data Sources Mix of high-quality text data: - FineWeb-Edu (80B tokens) - OpenWebMath (30B tokens) - DCLM-Baseline (30B tokens) - OpenMathReasoning (20B tokens) - OpenR1-Math (20B tokens) - ML-ArXiv (20B tokens) - Wikipedia (15B tokens) - PG-19 Books (15B tokens) - GSM8K Enhanced (15B tokens) - The Stack (10B tokens) ## Usage ### Streaming (Recommended for 32GB storage) ```python from datasets import load_dataset dataset = load_dataset("albertlungu/final-nous-corpus", split="train", streaming=True) for example in dataset: text = example["text"] # Tokenize and train... ``` ### With Streaming Dataloader ```python from src.data.streaming_dataset import create_dataloader dataloader = create_dataloader( repo_id="albertlungu/final-nous-corpus", batch_size=8, seq_length=4096, rank=0, world_size=4, ) for batch in dataloader: input_ids = batch["input_ids"] # [8, 4096] labels = batch["labels"] # Train... ``` ## License This dataset is a compilation of publicly available sources. Each component retains its original license. ## Citation If you use this dataset, please cite the original sources.