DeepSeek-R1-Distill-Qwen-1.5B-pts-dpo-pairs

# PTS DPO Dataset A Direct Preference Optimization (DPO) dataset created using the Pivotal Token Search (PTS) technique. ## Details - **Source:** Generated using the [PTS](https://github.com/codelion/pts) tool - **Model:** deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B ## Format Each example in the dataset consists of: - `prompt`: The context leading up to the pivotal token - `chosen`: The preferred token that increases success probability - `rejected`: The alternative token that decreases success probability - `metadata`: Additional information about the example ## Usage This dataset can be used for fine-tuning language models with Direct Preference Optimization (DPO). For a quick start, you can use our Google Colab notebook to fine-tune a model using this DPO dataset: [](https://colab.research.google.com/drive/1FggA9EQ1eFBjE0Qbsl0-EFzyWIxpdhlH?usp=sharing) ```python from datasets import load_dataset # Load the dataset from Hugging Face dataset = load_dataset("codelion/DeepSeek-R1-Distill-Qwen-1.5B-pts-dpo-pairs") # Use with your favorite DPO implementation # Example with TRL library: from trl import DPOTrainer trainer = DPOTrainer( model=model, args=training_args, beta=0.1, train_dataset=dataset, tokenizer=tokenizer, # ... other parameters ) trainer.train() ```