Magpie-DPO-100K-SML

This dataset is still under internal assessment. Please use it with caution!

To create this dataset, we first generate responses from the base model using URIAL as rejected. Then, we generate responses from 8B, 70B, and 405B models, and take the instruction-response pair with the highest reward as chosen.

### Other Magpie DPO Datasets
We observed that the following DPO datasets may have better performance after we burned a lot of GPU hours :)

|Model Name | Dataset | Type | Description |
|-------------|:-------|:-------|:-------|
| [Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | [Magpie-Air-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Air-DPO-100K-v0.1) | DPO | DPO dataset via Best-of-N sampling and rewards.
| [Llama 3 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | [Magpie-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-DPO-100K-v0.1) | DPO | DPO dataset via Best-of-N sampling and rewards.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1) | DPO | DPO dataset via Best-of-N sampling and rewards.

本数据集目前仍处于内部评估阶段，请谨慎使用！

为构建本数据集，我们首先使用URIAL生成基础模型的回复作为拒绝样本（rejected）；随后，我们分别从8B、70B及405B参数的模型中生成回复，并选取奖励分值最高的指令-回复对作为优选样本（chosen）。

### 其他Magpie DPO数据集
我们在耗费大量GPU计算资源后发现，以下直接偏好优化（Direct Preference Optimization，DPO）数据集可实现更优性能:)

| 模型名称 | 数据集 | 类型 | 描述 |
|-------------|:-------|:-------|:-------|
| [Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | [Magpie-Air-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Air-DPO-100K-v0.1) | 直接偏好优化（Direct Preference Optimization，DPO） | 基于最优N采样与奖励机制构建的DPO数据集。
| [Llama 3 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | [Magpie-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-DPO-100K-v0.1) | DPO | 基于最优N采样与奖励机制构建的DPO数据集。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1) | DPO | 基于最优N采样与奖励机制构建的DPO数据集。

搜集汇总

背景与挑战

背景概述

Magpie-DPO-100K-SML是一个基于DPO（直接偏好优化）的数据集，通过基础模型生成响应并以URIAL作为拒绝样本，然后从多个规模模型中选取奖励最高的指令-响应对构建而成。该数据集目前仍处于内部评估阶段，建议用户谨慎使用，其许可证为Apache 2.0，发布于2025年1月16日，大小约为667.29MB。