Infinity-Preference

## Infinity-Preference

The focus of human preferences varies from task to task. Therefore, Infinity-Preference attempts to adjust preference attribute weights on each task based on (Infinity Instruct's)[https://huggingface.co/datasets/BAAI/Infinity-Instruct] capability labelling system. This version contains 59438 evenly sampled instructions from Infinity-Instruct's instruction set for each task type. Each instruction is accompanied by a preference pair sampled from Gemma-2-9B-IT. This preference pair is annotated by the task-specific preference attribute weights and ArmoRM. You can also use Infinity-Preference to produce on-policy data for more models. We will be releasing code for building task-specific weights soon.


## News

- 🔥🔥🔥[2024/08/29] We release the first [Simpo](https://github.com/princeton-nlp/SimPO) model finetuned on Infinity-Preference, [Gemma2-9B-IT-Simpo-Infinity-Preference](https://huggingface.co/BAAI/Gemma2-9B-IT-Simpo-Infinity-Preference/settings). It achieves 73.4% LC win-rate on AlpacaEval 2.0 and 58.1% win-rate on Arena-Hard.

## Diversity
We did a visual analysis of the task distribution of Infinity-Preference as well as open source datasets and review lists based on the embedding of [BGE](https://huggingface.co/BAAI/bge-large-en-v1.5). Infinity-preference is the part in red, and you can see that Infinity-preference basically covers the task distribution of mainstream lists and datasets. It shows that Infinity-Preference is helpful to support the full modelling of each task type.


## Disclaimer

The resources, including code, data, and model weights, associated with this project are restricted for academic research purposes only and cannot be used for commercial purposes. The content produced by any version of Infinity-Preference is influenced by uncontrollable variables such as randomness, and therefore, the accuracy of the output cannot be guaranteed by this project. This project does not accept any legal liability for the content of the model output, nor does it assume responsibility for any losses incurred due to the use of associated resources and output results.

# Infinity-Preference
人类偏好的侧重点因任务而异。因此，Infinity-Preference 尝试基于Infinity Instruct的能力标注体系（详见https://huggingface.co/datasets/BAAI/Infinity-Instruct），为每类任务调整偏好属性权重。本版本从Infinity-Instruct的指令集中为每一类任务均匀采样了59438条指令，每条指令均附带从Gemma-2-9B-IT中采样得到的偏好对。该偏好对通过任务专属偏好属性权重与ArmoRM完成标注。您还可借助Infinity-Preference为更多模型生成同策略（on-policy）数据，我们即将发布用于构建任务专属权重的代码。


## 新闻动态
- 🔥🔥🔥[2024/08/29] 我们发布了首个基于Infinity-Preference微调的Simpo模型（详见https://github.com/princeton-nlp/SimPO），即Gemma2-9B-IT-Simpo-Infinity-Preference（模型链接：https://huggingface.co/BAAI/Gemma2-9B-IT-Simpo-Infinity-Preference/settings）。该模型在AlpacaEval 2.0上的LC胜率达73.4%，在Arena-Hard上的胜率为58.1%。

## 多样性分析
我们基于BGE嵌入向量（详见https://huggingface.co/BAAI/bge-large-en-v1.5），对Infinity-Preference及其他开源数据集与评测榜单的任务分布开展了可视化分析。其中Infinity-Preference以红色区块呈现，可见其基本覆盖了主流榜单与数据集的任务分布，表明该数据集有助于实现各类任务类型的完整建模。


## 免责声明
本项目关联的代码、数据与模型权重等资源仅可用于学术研究，不得用于商业用途。任何版本的Infinity-Preference生成的内容均受随机性等不可控变量影响，本项目无法保证输出内容的准确性。本项目不对模型输出内容承担任何法律责任，亦不对因使用本项目关联资源与输出结果所导致的任何损失承担责任。