sutra-magpie-sft

# Sutra Magpie SFT Dataset A high-quality dataset of **20,682 instruction-response pairs** for supervised fine-tuning (SFT) of language models. Generated using seed prompts from the Sutra framework with Magpie-style response generation. ## Dataset Description This dataset provides diverse, high-quality instruction-response pairs suitable for training instruction-following language models. ### Generation Method 1. **Seed Prompts**: Started with 30K diverse seed prompts from [codelion/sutra-30k-seeds](https://huggingface.co/datasets/codelion/sutra-30k-seeds) 2. **Response Generation**: Generated responses using Magpie-style approach with an instruction-tuned LLM 3. **Quality Filtering**: Removed incomplete responses, duplicates, and formatting issues ### Quality Assurance The dataset underwent rigorous cleaning: - Removed 9,615 incomplete responses (asking for missing context) - Removed 47 duplicate responses - Fixed 17,917 formatting issues (double spaces, trailing spaces) - All entries validated for proper instruction-response structure ### Statistics | Metric | Value | |--------|-------| | Total Entries | 20,682 | | Unique Instructions | 100% | | Unique Responses | 100% | **Instruction Length:** | Min | Max | Average | |-----|-----|---------| | 19 chars | 500 chars | 316 chars | **Response Length:** | Min | Max | Average | |-----|-----|---------| | 6 chars | 3,141 chars | 1,851 chars | ## Dataset Structure ```json { "instruction": "What are some effective strategies for improving time management?", "response": "Here are several proven strategies for better time management:\n\n1. **Prioritize tasks**...", "full_text": "User: What are some effective strategies...\n\nAssistant: Here are several proven strategies...", "seed_index": 42 } ``` ### Fields | Field | Description | |-------|-------------| | `instruction` | The user's instruction/question | | `response` | The assistant's response | | `full_text` | Combined user-assistant format | | `seed_index` | Reference to original seed prompt | ## Usage ### Loading the Dataset ```python from datasets import load_dataset dataset = load_dataset("codelion/sutra-magpie-sft", split="train") print(f"Loaded {len(dataset)} instruction-response pairs") # Access an example print(f"Instruction: {dataset[0]['instruction']}") print(f"Response: {dataset[0]['response'][:200]}...") ``` ### Training with Transformers ```python from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments from trl import SFTTrainer # Load model and tokenizer model = AutoModelForCausalLM.from_pretrained("your-base-model") tokenizer = AutoTokenizer.from_pretrained("your-base-model") # Format for training def format_prompt(example): return f"User: {example['instruction']}\n\nAssistant: {example['response']}" # Train trainer = SFTTrainer( model=model, train_dataset=dataset, formatting_func=format_prompt, max_seq_length=2048, ) trainer.train() ``` ## Content Coverage The dataset covers diverse capabilities: - **Chat**: Casual conversation, advice, explanations, roleplay, creative writing - **Reasoning**: Mathematical, logical, causal, counterfactual, multi-step problems - **Long Context**: Summarization, document QA, analysis, information synthesis - **Tools**: Function calling, API usage, search queries, calculations ## Related Datasets - [sutra-100M](https://huggingface.co/datasets/codelion/sutra-100M): 100M token pretraining dataset - [sutra-10M](https://huggingface.co/datasets/codelion/sutra-10M): 10M token pretraining dataset - [sutra-30k-seeds](https://huggingface.co/datasets/codelion/sutra-30k-seeds): Instruction prompts for post-training ## Citation ```bibtex @article{sharma2026sutra, title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens}, author={Sharma, Asankhaya}, year={2026}, url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens} } ``` ## License Apache 2.0