sutra-100M

# Sutra 100M Pretraining Dataset A high-quality synthetic pedagogical dataset designed for LLM pretraining, containing 70,435 educational entries totaling approximately 100 million tokens. ## Dataset Description This dataset was generated using the Sutra framework, which creates structured educational content optimized for language model pretraining. Each entry is designed to maximize learning efficiency through: - **Clear pedagogical structure**: Content follows proven educational patterns - **Cross-domain connections**: Concepts are linked across disciplines - **Varied complexity levels**: From foundational (level 1) to advanced (level 10) - **Quality-controlled generation**: All entries meet minimum quality thresholds ## Dataset Statistics | Metric | Value | |--------|-------| | Total Entries | 70,435 | | Total Tokens | 100,009,696 | | Avg Tokens/Entry | 1,419 | | Quality Score Range | 60.0 - 70.0 | | Average Quality Score | 65.6 | ### Domain Distribution | Domain | Count | Percentage | |--------|-------|------------| | Science | 23,883 | 33.9% | | Mathematics | 21,058 | 29.9% | | Programming & Systems | 11,451 | 16.3% | | Logic & Reasoning | 4,449 | 6.3% | | Engineering | 4,163 | 5.9% | | Language & Communication | 1,688 | 2.4% | | Applied Fields | 1,413 | 2.0% | | Social Sciences | 1,254 | 1.8% | | Humanities | 1,076 | 1.5% | ### Content Type Distribution | Content Type | Count | Percentage | |--------------|-------|------------| | Reasoning Demonstration | 18,225 | 25.9% | | Concept Introduction | 15,583 | 22.1% | | Cross-Domain Bridge | 15,090 | 21.4% | | Meta-Learning | 12,268 | 17.4% | | Synthesis | 9,269 | 13.2% | ### Complexity Distribution | Level | Count | Percentage | |-------|-------|------------| | Level 1-3 | 967 | 1.4% | | Level 4-5 | 2,935 | 4.2% | | Level 6 | 4,791 | 6.8% | | Level 7 | 15,961 | 22.7% | | Level 8 | 23,401 | 33.2% | | Level 9 | 17,888 | 25.4% | | Level 10 | 4,492 | 6.4% | ## Data Fields Each entry contains the following fields: | Field | Description | |-------|-------------| | `id` | Unique identifier for the entry | | `concept_name` | The concept being taught | | `domain` | Primary knowledge domain | | `content_type` | Type of pedagogical content | | `text` | The main educational content | | `quality_score` | Quality assessment score (0-100) | | `information_density` | Measure of information per token | | `complexity_level` | Difficulty level (1-10) | | `prerequisites` | Required prior knowledge | | `builds_to` | Advanced concepts this enables | | `cross_domain_connections` | Links to other domains | | `token_count` | Number of tokens in the entry | | `quality_assessment` | Detailed quality breakdown | ## Generation Details - **Generator**: Sutra Framework - **Quality Threshold**: Minimum score of 60.0 - **Generation Date**: June 2025 ## Intended Use This dataset is designed for: - **LLM Pretraining**: High-quality educational content for foundational model training - **Domain-specific fine-tuning**: Subset by domain for specialized training - **Educational AI research**: Studying pedagogical content generation ## Quality Assurance All entries underwent multi-dimensional quality assessment including: - Clarity and readability - Information density - Pedagogical structure - Reasoning completeness - Practical utility - Connection richness ## Related Datasets - [sutra-10M](https://huggingface.co/datasets/codelion/sutra-10M): 10M token pretraining dataset - [sutra-30k-seeds](https://huggingface.co/datasets/codelion/sutra-30k-seeds): Instruction prompts for post-training - [sutra-magpie-sft](https://huggingface.co/datasets/codelion/sutra-magpie-sft): SFT dataset generated from seed prompts ## Citation ```bibtex @article{sharma2026sutra, title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens}, author={Sharma, Asankhaya}, year={2026}, url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens} } ``` ## License Apache 2.0