olmo-2-0325-32b-preference-mix

# OLMo 2 0325 32B Preference Mixture

*Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.*

This mix is made up of the following on-policy preference datasets generated using a synthetic data generation pipeline similar to Tulu 3:

- Reused prompts from the SFT mix (allenai/sft_v3.9_used_off_policy_prompts-olmo32)
- Reused prompts from the SFT mix filtered for instruction-following (allenai/IF_Taxonomy-olmo32)
- Reused prompts in SFT subsampled from WildChat (allenai/wildchat_v3.9_used_on_policy_prompts-olmo32 and allenai/WildChat-prefs-280824-olmo32)
- Cleaned version of Ultrafeedback without ShareGPT and TruthfulQA instances (allenai/uf_cleaned-olmo32)
- Prompts from WildChat that wasn't used in the SFT mix (allenai/wildchat_v3.9_unused_off_policy_prompts-olmo32)
- Prompts from DaringAnteater (allenai/DaringAnteater-preferences-olmo32)
- Tulu3 Personas with instruction-following (allenai/tulu-3-pref-personas-instruction-following) (GPT4o, not on policy)

This preference mixture used for DPO on our the [OLMo-2-0325-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT) checkpoint to obtain [OLMo-2-0325-32B-DPO](https://huggingface.co/allenai/OLMo-2-0325-32B-DPO).

It contains 377.9k generation pairs obtained using the following models:
- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) (Apache 2.0)
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) (Apache 2.0)
- [Tulu 2 7B](https://huggingface.co/allenai/tulu-2-7b) (Ai2 ImpACT Low Risk License)
- [Tulu 2 13B](https://huggingface.co/allenai/tulu-2-13b) (Ai2 ImpACT Low Risk License)
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat) (Apache 2.0)
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) (Apache 2.0)
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat) (CC-BY-SA-4.0)
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat) (CC-BY-SA-4.0)
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct) (Apache 2.0)
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) (Apache 2.0)
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) (Apache 2.0)
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) (Apache 2.0)
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) and [GPT-4o](https://openai.com/index/hello-gpt-4o/) (Outputs produced by GPT-4 are subject to OpenAI's [terms of use](https://openai.com/policies/row-terms-of-use))
- [Microsoft Phi 3 Mini 128k Instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (MIT)
- [Microsoft Phi 3.5 Mini Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) (MIT)
- [NuMind NuExtract v1.5](https://huggingface.co/numind/NuExtract-1.5) (MIT)

The data was [filtered](https://github.com/allenai/open-instruct/blob/ab437221f87627a2b39280003be2fc0eff9a8dcb/scripts/data/filtering_and_updates/filter_cutoff_date.py) to remove instances where the chosen response had mentions of a date cutoff.

## License
This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use). This dataset includes output data generated from third party models that are subject to separate terms governing their use.

# OLMo 2 0325 32B 偏好混合数据集

*请注意，本数据集集合采用ODC-BY-1.0许可证授权；数据子集适用不同的许可证条款。本数据集的部分内容为非商用性质。我们将此混合数据集作为研究用制品发布。*

本混合数据集由以下基于与Tulu 3类似的合成数据生成流水线构建的在线策略（on-policy）偏好数据集组成：
- 复用自监督微调（Supervised Fine-Tuning, SFT）混合数据集的离线策略（off-policy）提示词（allenai/sft_v3.9_used_off_policy_prompts-olmo32）
- 经指令遵循任务过滤后的监督微调（Supervised Fine-Tuning, SFT）混合数据集复用提示词（allenai/IF_Taxonomy-olmo32）
- 从WildChat中采样得到的监督微调（Supervised Fine-Tuning, SFT）数据集内的复用提示词（包含allenai/wildchat_v3.9_used_on_policy_prompts-olmo32与allenai/WildChat-prefs-280824-olmo32）
- 移除了ShareGPT与TruthfulQA样本的清洗版Ultrafeedback数据集（allenai/uf_cleaned-olmo32）
- 未在监督微调（Supervised Fine-Tuning, SFT）混合数据集中使用的WildChat离线策略（off-policy）提示词（allenai/wildchat_v3.9_unused_off_policy_prompts-olmo32）
- DaringAnteater偏好数据集的提示词（allenai/DaringAnteater-preferences-olmo32）
- 带指令遵循任务的Tulu3角色数据集（allenai/tulu-3-pref-personas-instruction-following）（由GPT-4o生成，属于非在线策略数据）

本偏好混合数据集用于对[OLMo-2-0325-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT)模型检查点执行直接偏好优化（Direct Preference Optimization, DPO），以得到[OLMo-2-0325-32B-DPO](https://huggingface.co/allenai/OLMo-2-0325-32B-DPO)。

本数据集包含37.79万条生成样本对，其生成所用模型如下：
- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)（采用Apache 2.0许可证）
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)（采用Apache 2.0许可证）
- [Tulu 2 7B](https://huggingface.co/allenai/tulu-2-7b)（采用Ai2 ImpACT 低风险许可证）
- [Tulu 2 13B](https://huggingface.co/allenai/tulu-2-13b)（采用Ai2 ImpACT 低风险许可证）
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat)（采用Apache 2.0许可证）
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)（采用Apache 2.0许可证）
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat)（采用CC-BY-SA-4.0许可证）
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat)（采用CC-BY-SA-4.0许可证）
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it)（Gemma的使用需遵守[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms)中规定的Gemma使用条款）
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it)（Gemma的使用需遵守[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms)中规定的Gemma使用条款）
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用。如需获取商用许可，请参照模型卡片中的说明操作）
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用。如需获取商用许可，请参照模型卡片中的说明操作）
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用。如需获取商用许可，请参照模型卡片中的说明操作）
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct)（采用Apache 2.0许可证）
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct)（采用Apache 2.0许可证）
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)（采用Apache 2.0许可证）
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7b-Instruct)（采用Apache 2.0许可证）
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/)与[GPT-4o](https://openai.com/index/hello-gpt-4o/)（GPT-4生成的输出需遵守OpenAI的[使用条款](https://openai.com/policies/row-terms-of-use)）
- [Microsoft Phi 3 Mini 128k Instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)（采用MIT许可证）
- [Microsoft Phi 3.5 Mini Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)（采用MIT许可证）
- [NuMind NuExtract v1.5](https://huggingface.co/numind/NuExtract-1.5)（采用MIT许可证）

本数据集已通过[过滤脚本](https://github.com/allenai/open-instruct/blob/ab437221f87627a2b39280003be2fc0eff9a8dcb/scripts/data/filtering_and_updates/filter_cutoff_date.py)移除了所有被选中回复中提及日期截止限制的样本。

## 许可证
本数据集采用ODC-BY许可证授权。其旨在符合艾伦人工智能研究所（Allen Institute for AI, Ai2）的[负责任使用指南](https://allenai.org/responsible-use)，仅用于研究与教育用途。本数据集包含由第三方模型生成的输出数据，此类数据需遵守对应模型的单独使用条款。