olmo-2-0425-1b-preference-mix

# OLMo 2 0425 1B Preference Mixture

*Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.*

This mix is made up of the following on-policy preference datasets generated using a synthetic data generation pipeline similar to Tulu 3:

- Reused prompts from the SFT mix (allenai/sft_v3.9_used_off_policy_prompts-olmo32)
- Reused prompts from the SFT mix filtered for instruction-following (allenai/wildchat_IF_Taxonomy-olmo32)
- Reused prompts in SFT subsampled from WildChat (allenai/wildchat_v3.9_used_on_policy_prompts-olmo32 and allenai/WildChat-prefs-280824-olmo32)
- Cleaned version of Ultrafeedback without ShareGPT and TruthfulQA instances (allenai/uf_cleaned-olmo32)
- Prompts from WildChat that wasn't used in the SFT mix (allenai/wildchat_v3.9_unused_off_policy_prompts-olmo32)
- Prompts from DaringAnteater (allenai/DaringAnteater-preferences-olmo32)
- Tulu3 Personas with instruction-following (allenai/tulu-3-pref-personas-instruction-following) (GPT4o, not on policy)

This preference mixture used for DPO on our the [OLMo-2-0325-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT) checkpoint to obtain [OLMo-2-0325-32B-DPO](https://huggingface.co/allenai/OLMo-2-0325-32B-DPO).

It contains 377.9k generation pairs obtained using the following models:
- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) (Apache 2.0)
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) (Apache 2.0)
- [Tulu 2 7B](https://huggingface.co/allenai/tulu-2-7b) (Ai2 ImpACT Low Risk License)
- [Tulu 2 13B](https://huggingface.co/allenai/tulu-2-13b) (Ai2 ImpACT Low Risk License)
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat) (Apache 2.0)
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) (Apache 2.0)
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat) (CC-BY-SA-4.0)
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat) (CC-BY-SA-4.0)
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct) (Apache 2.0)
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) (Apache 2.0)
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) (Apache 2.0)
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) (Apache 2.0)
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) and [GPT-4o](https://openai.com/index/hello-gpt-4o/) (Outputs produced by GPT-4 are subject to OpenAI's [terms of use](https://openai.com/policies/row-terms-of-use))
- [Microsoft Phi 3 Mini 128k Instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (MIT)
- [Microsoft Phi 3.5 Mini Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) (MIT)
- [NuMind NuExtract v1.5](https://huggingface.co/numind/NuExtract-1.5) (MIT)

The data was [filtered](https://github.com/allenai/open-instruct/blob/ab437221f87627a2b39280003be2fc0eff9a8dcb/scripts/data/filtering_and_updates/filter_cutoff_date.py) to remove instances where the chosen response had mentions of a date cutoff.

## License
This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use). This dataset includes output data generated from third party models that are subject to separate terms governing their use.

# OLMo 2 0425 1B 偏好混合数据集

> 注意本数据集集合采用ODC-BY-1.0协议授权；数据集的子集适用不同的许可协议。本数据集包含部分非商用内容，本混合数据集仅作为研究成果发布。

本混合数据集由以下基于类Tulu 3的合成数据生成流水线构建的在线策略（on-policy）偏好数据集组成：
- 从监督微调（Supervised Fine-Tuning，简称SFT）混合数据集复用的离线策略提示词（allenai/sft_v3.9_used_off_policy_prompts-olmo32）
- 经过指令遵循任务过滤的SFT混合数据集复用提示词（allenai/wildchat_IF_Taxonomy-olmo32）
- 从WildChat中采样得到的SFT复用提示词，包含allenai/wildchat_v3.9_used_on_policy_prompts-olmo32与allenai/WildChat-prefs-280824-olmo32两个子集
- 移除ShareGPT与TruthfulQA样本后的清理版Ultrafeedback数据集（allenai/uf_cleaned-olmo32）
- 未在SFT混合数据集中使用的WildChat离线策略提示词（allenai/wildchat_v3.9_unused_off_policy_prompts-olmo32）
- DaringAnteater偏好数据集的提示词（allenai/DaringAnteater-preferences-olmo32）
- 带有指令遵循设定的Tulu3人设偏好数据集（allenai/tulu-3-pref-personas-instruction-following）（由GPT-4o生成，非在线策略数据）

本偏好混合数据集用于在[OLMo-2-0325-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT)模型检查点上进行直接偏好优化（Direct Preference Optimization，简称DPO）训练，以得到[OLMo-2-0325-32B-DPO](https://huggingface.co/allenai/OLMo-2-0325-32B-DPO)模型。

本数据集共包含377.9k个生成样本对，所使用的模型及对应许可如下：
- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)（Apache 2.0协议）
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)（Apache 2.0协议）
- [Tulu 2 7B](https://huggingface.co/allenai/tulu-2-7b)（艾伦研究所ImpACT低风险许可协议）
- [Tulu 2 13B](https://huggingface.co/allenai/tulu-2-13b)（艾伦研究所ImpACT低风险许可协议）
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat)（Apache 2.0协议）
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)（Apache 2.0协议）
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat)（CC-BY-SA-4.0协议）
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat)（CC-BY-SA-4.0协议）
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it)（Gemma模型需遵循[Gemma使用条款](https://ai.google.dev/gemma/terms)）
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it)（Gemma模型需遵循[Gemma使用条款](https://ai.google.dev/gemma/terms)）
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用；如需商用许可，请参照模型卡片说明获取）
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用；如需商用许可，请参照模型卡片说明获取）
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商用；如需商用许可，请参照模型卡片说明获取）
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct)（Apache 2.0协议）
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct)（Apache 2.0协议）
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)（Apache 2.0协议）
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)（Apache 2.0协议）
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/)与[GPT-4o](https://openai.com/index/hello-gpt-4o/)（GPT-4生成的输出内容受OpenAI的[使用条款](https://openai.com/policies/row-terms-of-use)约束）
- [Microsoft Phi 3 Mini 128k Instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)（MIT协议）
- [Microsoft Phi 3.5 Mini Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)（MIT协议）
- [NuMind NuExtract v1.5](https://huggingface.co/numind/NuExtract-1.5)（MIT协议）

本数据集已通过[过滤脚本](https://github.com/allenai/open-instruct/blob/ab437221f87627a2b39280003be2fc0eff9a8dcb/scripts/data/filtering_and_updates/filter_cutoff_date.py)移除了所有首选回复中提及日期截止限制的样本。

## 许可协议
本数据集采用ODC-BY协议授权，仅用于研究与教育用途，需遵循艾伦研究所的[负责任使用指南](https://allenai.org/responsible-use)。本数据集包含由第三方模型生成的输出内容，该部分内容受对应模型自身的使用条款约束。