fnlp/hh-rlhf-strength-cleaned
收藏资源简介：
--- license: apache-2.0 --- # Dataset Card for hh-rlhf-strength-cleaned **Other Language Versions: [English](README.md), [中文](README_zh.md).** ## Dataset Description In the paper titled "[Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://arxiv.org/abs/2401.06080)" we measured the preference strength of each preference pair in the [hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf "https://huggingface.co/datasets/Anthropic/hh-rlhf") dataset through model ensemble and annotated the valid set with GPT-4. In this repository, we provide: 1. Metadata of preference strength for both the training and valid sets. 2. GPT-4 annotations on the valid set. We mixed the hh-rlhf dataset and redivided it into a training set (151k) and a valid set (17k) with a ratio of 9:1. ## Field Description | Field Name | Field Description | Remarks | | --------------------------- | ------------------------------------------------------------------------------ | ------------------------------------- | | chosen | Same as the hh-rlhf dataset. The last line represents the chosen response, and the preceding lines constitute the dialogue history | Type is a list. The dialogue history for both chosen and rejected responses is the same | | rejected | Same as the hh-rlhf dataset. The last line represents the chosen response, and the preceding lines constitute the dialogue history | Type is a list. The dialogue history for both chosen and rejected responses is the same | | GPT4 label | GPT-4 annotation for preference pairs; 1 indicates GPT-4 prefers chosen, 0 indicates GPT-4 prefers rejected, and -1 indicates that the label does not exist | Only present in the valid set | | mean preference difference | Metric measuring preference strength as discussed in the paper; absolute value indicates the magnitude, and positive/negative indicates preference for chosen or rejected, respectively | Average of preference strengths across N models | | std preference difference | Metric measuring uncertainty in preference strength, representing the standard deviation among preference strengths from different models | Standard deviation of preference strengths across N models | | chosen score list | List of scores given by N models for the chosen option in each preference pair | Type is a list, each element represents the score given by a single model | | rejected score list | List of scores given by N models for the rejected option in each preference pair | Type is a list, each element represents the score given by a single model |
--- license: apache-2.0 --- # hh-rlhf-strength-cleaned 数据集卡片 **其他语言版本：[英文](README.md)、[中文](README_zh.md)。** ## 数据集概述 在题为《大语言模型RLHF的奥秘 第二部分：奖励建模》（*Secrets of RLHF in Large Language Models Part II: Reward Modeling*，arXiv:2401.06080）的论文中，我们通过模型集成的方式对[hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)数据集中的每一组偏好对的偏好强度进行了量化，并使用GPT-4（Generative Pre-trained Transformer 4）对验证集进行了标注。本仓库提供以下内容： 1. 训练集与验证集的偏好强度元数据 2. 验证集的GPT-4标注结果 我们对原始hh-rlhf数据集进行了混合重组，并按照9:1的比例划分为训练集（151k条）与验证集（17k条）。 ## 字段说明 | 字段名 | 字段描述 | 备注 | | -------------------------- | ---------------------------------------------------------------------- | ------------------------------------- | | chosen | 与原始hh-rlhf数据集一致，列表最后一行为选中的回复，其余行构成对话历史 | 数据类型为列表，选中回复与落选回复共享同一对话历史 | | rejected | 与原始hh-rlhf数据集一致，列表最后一行为落选的回复，其余行构成对话历史 | 数据类型为列表，选中回复与落选回复共享同一对话历史 | | GPT4 label | 针对偏好对的GPT-4标注标签；1表示GPT-4偏好选中回复，0表示GPT-4偏好落选回复，-1表示标签不存在 | 仅在验证集中出现 | | mean preference difference | 如论文所述，用于衡量偏好强度的指标，其绝对值表示强度大小，正负分别表示对选中回复或落选回复的偏好 | 为N个模型计算出的偏好强度的平均值 | | std preference difference | 用于衡量偏好强度的不确定性，代表不同模型计算出的偏好强度的标准差 | 为N个模型计算出的偏好强度的标准差 | | chosen score list | 每组偏好对中，N个模型为选中回复给出的得分列表 | 数据类型为列表，每个元素对应单个模型给出的得分 | | rejected score list | 每组偏好对中，N个模型为落选回复给出的得分列表 | 数据类型为列表，每个元素对应单个模型给出的得分 |
数据集语言版本
- 英语
- 中文
