Magpie-Align/MagpieLM-DPO-Data-v0.1

---
dataset_info:
features:
- name: uuid
dtype: string
- name: instruction
dtype: string
- name: chosen
list:
- name: content
dtype: string
- name: role
dtype: string
- name: rejected
list:
- name: content
dtype: string
- name: role
dtype: string
splits:
- name: train
num_bytes: 1118772760
num_examples: 196000
- name: test
num_bytes: 22814240
num_examples: 4000
download_size: 616500602
dataset_size: 1141587000
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: test
path: data/test-*
license: llama3.1
language:
- en
size_categories:
- 100K<n<1M
---


Project Web: [https://magpie-align.github.io/](https://magpie-align.github.io/)

Arxiv Technical Report: [https://arxiv.org/abs/2406.08464](https://arxiv.org/abs/2406.08464)

Codes: [https://github.com/magpie-align/magpie](https://github.com/magpie-align/magpie)

## 🧐 Dataset Details
The Magpie Team generates this dataset for direct preference optimization. This dataset was used to train [Magpie-Align/MagpieLM-4B-Chat-v0.1](https://huggingface.co/Magpie-Align/MagpieLM-4B-Chat-v0.1).

This dataset is a combination of two datasets:
- Half of the dataset (100K) is from [Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1/tree/main).
- Another half of the dataset (100K) uses the instructions from [Magpie-Align/Magpie-Air-DPO-100K-v0.1](https://huggingface.co/datasets/Magpie-Align/Magpie-Air-DPO-100K-v0.1), then generated responses using [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) 5 times for each instruction, using a temperature of 0.8. We then annotated RM scores using [RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1), labeling the response with the highest RM score as the chosen response, and the one with the lowest RM score as the rejected response.

Why Magpie 💜 Gemma-2-9B? Take a look at our latest paper: [Stronger Models are NOT Stronger Teachers for Instruction Tuning](https://huggingface.co/papers/2411.07133). We found that stronger models are not always stronger teachers for instruction tuning!

**License**: Please follow [Meta Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) and [Gemma License](https://www.kaggle.com/models/google/gemma/license/).

## 📚 Citation

If you find the model, data, or code useful, please cite our paper:
```
@article{xu2024magpie,
title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing},
author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
year={2024},
eprint={2406.08464},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

Please also cite the reward model for creating preference datasets:

ArmoRM paper:
```
@article{wang2024interpretable,
title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
journal={arXiv preprint arXiv:2406.12845},
year={2024}
}

@article{xu2024stronger,
title={Stronger Models are NOT Stronger Teachers for Instruction Tuning},
author={Xu, Zhangchen and Jiang, Fengqing and Niu, Luyao and Lin, Bill Yuchen and Poovendran, Radha},
journal={arXiv preprint arXiv:2411.07133},
year={2024}
}
```

**Contact**

Questions? Contact:
- [Zhangchen Xu](https://zhangchenxu.com/) [zxu9 at uw dot edu], and
- [Bill Yuchen Lin](https://yuchenlin.xyz/) [yuchenlin1995 at gmail dot com]