sutra-30k-seeds

# Sutra 30K Seeds A curated dataset of **30,320 diverse instruction prompts** designed for generating high-quality SFT (Supervised Fine-Tuning) datasets. These seeds serve as the foundation for creating instruction-response pairs for post-training language models. ## Dataset Description This dataset contains seed prompts across 4 primary capabilities and 18 sub-capabilities, designed to cover the core competencies needed for instruction-following models. ### Generation Method Seeds were generated using the Sutra framework: 1. **Capability Graph**: A knowledge graph of 18 sub-capabilities across 4 main capabilities 2. **LLM-based Variation**: Each seed is generated by prompting an LLM with capability descriptions and example prompts 3. **Quality Filtering**: Prompts are filtered for length (19-500 chars), deduplicated, and cleaned 4. **Cross-capability Mixing**: 44.8% of seeds combine multiple capabilities for complex instructions ### Statistics | Metric | Value | |--------|-------| | Total Seeds | 30,320 | | Unique Prompts | 100% | | Avg Prompt Length | 319 chars | | Min/Max Length | 19 / 500 chars | ### Capability Distribution | Capability | Count | Percentage | |------------|-------|------------| | Chat | 11,367 | 37.5% | | Reasoning | 9,562 | 31.5% | | Long Context | 6,665 | 22.0% | | Tools | 2,726 | 9.0% | ### Complexity Distribution | Complexity | Count | Percentage | |------------|-------|------------| | Advanced | 18,810 | 62.0% | | Intermediate | 8,389 | 27.7% | | Basic | 3,121 | 10.3% | ### Sub-capabilities The dataset covers 18 sub-capabilities: **Chat**: casual_conversation, advice_giving, explanation, roleplay, creative_writing **Reasoning**: mathematical_reasoning, logical_deduction, causal_reasoning, counterfactual, multi_step_problem **Long Context**: summarization, qa_over_documents, analysis, information_synthesis **Tools**: function_calling, api_usage, search_queries, calculation_requests **Cross-capability**: 13,590 seeds (44.8%) combine multiple capabilities ## Dataset Structure ```json { "seed_id": "seed_00001", "prompt": "What are some effective strategies for...", "capability": "chat", "sub_capability": "advice_giving", "complexity": "intermediate", "estimated_response_tokens": 300, "cross_capabilities": [], "metadata": { "generated_at": "2024-...", "node_description": "Providing helpful advice...", "complexity_score": 0.65 } } ``` ## Usage ### Loading the Dataset ```python from datasets import load_dataset dataset = load_dataset("codelion/sutra-30k-seeds", split="train") print(f"Loaded {len(dataset)} seed prompts") # Sample a prompt print(dataset[0]['prompt']) ``` ### Generating SFT Data (Magpie-style) These seeds are designed to be used with the Magpie approach for generating instruction-response pairs: ```python # Use each seed as the user turn, generate assistant response for seed in dataset: user_prompt = seed['prompt'] # Generate response using your model response = model.generate(user_prompt) # Create instruction-response pair ``` ### Related Datasets - [sutra-100M](https://huggingface.co/datasets/codelion/sutra-100M): 100M token pretraining dataset - [sutra-10M](https://huggingface.co/datasets/codelion/sutra-10M): 10M token pretraining dataset - [sutra-magpie-sft](https://huggingface.co/datasets/codelion/sutra-magpie-sft): SFT dataset generated from seed prompts ## Citation ```bibtex @article{sharma2026sutra, title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens}, author={Sharma, Asankhaya}, year={2026}, url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens} } ``` ## License Apache 2.0