tulu-3-sft-reused-on-policy-8b

<img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu-3/Tulu3-logo.png" alt="Tulu3 banner" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

# Llama 3.1 Tulu 3 SFT reused (on-policy 8b)

*Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.*

This preference dataset is part of our Tulu 3 preference mixture:
it contains prompts from our [SFT mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) and it contains 19,444 generation pairs (some of which on-policy from: https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B) obtained using the following models:

- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) (Apache 2.0)
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) (Apache 2.0)
- [Tulu 2 7B](https://huggingface.co/allenai/tulu-2-7b) (Ai2 ImpACT Low Risk License)
- [Tulu 2 13B](https://huggingface.co/allenai/tulu-2-13b) (Ai2 ImpACT Low Risk License)
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat) (Apache 2.0)
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) (Apache 2.0)
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat) (CC-BY-SA-4.0)
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat) (CC-BY-SA-4.0)
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it) (Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms))
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat) (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct) (Apache 2.0)
- [Qwen2.5 72B Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) (Qwen is licensed under the Qwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.)
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) (Apache 2.0)
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) (Apache 2.0)
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) (Apache 2.0)
- [Llama 3.1 8B Instruct ](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) (this dataset was partially "Built with Llama" and is thus subject to the Llama 3.1 License)
- [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (this dataset was partially "Built with Llama" and is thus subject to the Llama 3.1 License)
- [Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B) (this dataset was partially "Built with Meta Llama 3" and is thus subject to the Llama 3 License)
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) and [GPT-4o](https://openai.com/index/hello-gpt-4o/) (Outputs produced by GPT-4 are subject to OpenAI's [terms of use](https://openai.com/policies/row-terms-of-use))
- [Claude 3.5 Sonnet](https://www.anthropic.com/news/claude-3-5-sonnet) (Outputs produced by Claude are subject to Anthropic [terms of service](https://www.anthropic.com/legal/commercial-terms) and [usage policy](https://www.anthropic.com/legal/aup))

## License
This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use). This dataset includes output data generated from third party models that are subject to separate terms governing their use.

<img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu-3/Tulu3-logo.png" alt="Tulu3 横幅" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

# Llama 3.1 图鲁3（Tulu 3）监督微调（Supervised Fine-Tuning，SFT）复用数据集（同策略（on-policy）8B）

*请注意，本合集采用ODC-BY-1.0许可证授权；数据子集适用不同的许可证条款。本数据集部分内容为非商业性质。我们将此混合数据集作为研究成果发布。*

此偏好数据集属于我们的图鲁3（Tulu 3）偏好混合数据集的一部分：其提示词源自我们的[监督微调（Supervised Fine-Tuning，SFT）混合数据集](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture)，并包含19,444组生成对（其中部分同策略（on-policy）生成对来自https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B），生成这些数据所使用的模型如下：

- [Mistral 7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)（采用Apache 2.0许可证）
- [Mistral Nemo Instruct 2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)（采用Apache 2.0许可证）
- [图鲁2（Tulu 2）7B](https://huggingface.co/allenai/tulu-2-7b)（采用Ai2 ImpACT低风险许可证）
- [图鲁2（Tulu 2）13B](https://huggingface.co/allenai/tulu-2-13b)（采用Ai2 ImpACT低风险许可证）
- [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat)（采用Apache 2.0许可证）
- [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)（采用Apache 2.0许可证）
- [MPT 30B Chat](https://huggingface.co/mosaicml/mpt-30b-chat)（采用CC-BY-SA-4.0许可证）
- [MPT 7B 8k Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat)（采用CC-BY-SA-4.0许可证）
- [Google Gemma 2 27B it](https://huggingface.co/google/gemma-2-27b-it)（Gemma的使用需遵循[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms)中公布的Gemma使用条款）
- [Google Gemma 2 9B it](https://huggingface.co/google/gemma-2-9b-it)（Gemma的使用需遵循[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms)中公布的Gemma使用条款）
- [InternLM2.5 20B](https://huggingface.co/internlm/internlm2_5-20b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商业使用。如需获取商业许可证，请参照模型卡片中的说明操作）
- [InternLM2.5 7B](https://huggingface.co/internlm/internlm2_5-7b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商业使用。如需获取商业许可证，请参照模型卡片中的说明操作）
- [InternLM2.5 1.8B](https://huggingface.co/internlm/internlm2_5-1_8b-chat)（InternLM权重完全开放用于学术研究，同时支持免费商业使用。如需获取商业许可证，请参照模型卡片中的说明操作）
- [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b-instruct)（采用Apache 2.0许可证）
- [Qwen2.5 72B Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct)（Qwen采用Qwen许可协议授权，版权归阿里巴巴云计算所有，保留所有权利）
- [Qwen2.5 32B Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct)（采用Apache 2.0许可证）
- [Qwen2.5 14B Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)（采用Apache 2.0许可证）
- [Qwen2.5 7B Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)（采用Apache 2.0许可证）
- [Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)（本数据集部分内容"Built with Llama"，因此需遵循Llama 3.1许可证条款）
- [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct)（本数据集部分内容"Built with Llama"，因此需遵循Llama 3.1许可证条款）
- [Llama 3 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B)（本数据集部分内容"Built with Meta Llama 3"，因此需遵循Llama 3许可证条款）
- [GPT-4 Turbo](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) 与 [GPT-4o](https://openai.com/index/hello-gpt-4o/)（GPT-4生成的输出内容需遵循OpenAI的[使用条款](https://openai.com/policies/row-terms-of-use)）
- [Claude 3.5 Sonnet](https://www.anthropic.com/news/claude-3-5-sonnet)（Claude生成的输出内容需遵循Anthropic的[服务条款](https://www.anthropic.com/legal/commercial-terms)与[使用政策](https://www.anthropic.com/legal/aup)）

## 许可证声明

本数据集采用ODC-BY许可证授权。其旨在遵循艾伦人工智能研究所（Allen AI）的[负责任使用指南](https://allenai.org/responsible-use)，供研究与教育用途。本数据集包含由第三方模型生成的输出数据，此类数据需遵循各自独立的使用条款。

搜集汇总

背景与挑战

背景概述

该偏好数据集是Tulu 3偏好混合的一部分，包含19,444个生成对，部分数据基于Llama-3.1-Tulu-3-8B模型生成，并整合了Mistral、Tulu 2、Yi等多种模型的输出。数据集采用ODC-BY许可证，旨在用于研究和教育用途。