StellaWLX/hh-rlhf

--- license: mit tags: - human-feedback --- # Dataset Card for HH-RLHF ## Dataset Summary This repository provides access to two different kinds of data: 1. Human preference data about helpfulness and harmlessness from [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2204.05862). These data are meant to train preference (or reward) models for subsequent RLHF training. These data are *not* meant for supervised training of dialogue agents. Training dialogue agents on these data is likely to lead to harmful models and this shold be avoided. 2. Human-generated and annotated red teaming dialogues from [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://www.anthropic.com/red_teaming.pdf). These data are meant to understand how crowdworkers red team models and what types of red team attacks are succesful or not. The data are *not* meant for fine-tuning or preference modeling (use the data above for preference modeling). These data are entire transcripts of conversations that are derived from the harmlessness preference modeling data described above, where only the chosen response is incorporated into the overall transcript. Furthermore, the transcripts are annotated with human and automated measurements of how harmful the overall dialogues are. **Disclaimer**: The data (especially the harmlessness preference data and the red team data) contain content that may be offensive or upsetting. Topics include, but are not limited to, discriminatory language and discussions of abuse, violence, self-harm, exploitation, and other potentially upsetting subject matter. Please only engage with the data in accordance with your own personal risk tolerance. The data are intended for research purposes, especially research that can make models *less* harmful. The views expressed in the data do not reflect the views of Anthropic or any of its employees. As mentioned above, these data are *not* intended for training dialogue agents as this will likely lead to harmful model behavior. Each of these datasets are described further below. ## Human preference data about helpfulness and harmlessness (PM Data) The data are described in the paper: [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2204.05862). If you find the data useful, please cite the paper. The data format is very simple -- each line of the jsonl files contains a pair of texts, one "chosen" and one "rejected". For **helpfulness**, the data are grouped into train/test splits in three tranches: from our base models (context-distilled 52B language models), via rejection sampling (mostly with best-of-16 sampling) against an early preference model, and a dataset sampled during our iterated "online" process. For **harmlessness**, the data are only collected for our base models, but otherwise formatted in the same way. Details about the data collection process and crowdworker population can be found in the paper, specifically in section 2 and appendix D. ## Red teaming data (not PM Data) The data are described in the paper: [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://www.anthropic.com/red_teaming.pdf). If you find the data useful, please cite the paper. At a high level these data can be examined to understand what succesful red team attempts look like. Details about the data and data collection procedures can be found in the Datasheet in the appendix of the paper. Each line of the jsonl file contains a dictionary with the following fields: - `transcript` a text transcript of a conversation between a human adversary (red team member) and an AI assistant - `min_harmlessness_score_transcript` a real value score of the harmlessness of the AI assistant (lower is more harmful) as obtained from a preference model - `num_params` number of parameters in the language model powering the AI assistant - `model_type` type of model powering the AI assistant - `rating` the red team member's rating of how successful they were at breaking the AI assistant (Likert scale, higher is more successful) - `task_description` a short text description written by the red team member about how they tried to red team the AI assistant - `task_description_harmlessness_score` a real value score of the harmlessness of the task description (lower is more harmful) as obtained from a preference model - `red_team_member_id` an arbitrary identifier of the red team member. one red team member can generate multiple red team attacks - `is_upworker` a binary indicator that is true if the red team member was from the crowd platform Upwork or false if they were from MTurk - `tags` a list of up to 6 tags per transcript. tags are short descriptions of the red team attempts generated by crowdworkers who reviewed red team data post-hoc. tags were only provided for a random sample of 1000 red team attempts for two of four model types. ## Usage Each of the above datasets is located in a separate sub-directory. To load an individual subset, use the `data_dir` argument of the `load_dataset()` function as follows: ```python from datasets import load_dataset # Load all helpfulness/harmless subsets (share the same schema) dataset = load_dataset("Anthropic/hh-rlhf") # Load one of the harmless subsets dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base") # Load the red teaming subset dataset = load_dataset("Anthropic/hh-rlhf", data_dir="red-team-attempts") ``` ## Contact The original authors host this dataset on GitHub here: https://github.com/anthropics/hh-rlhf You can submit inquiries to: [e-mail]--- license: MIT tags: - 人类反馈（human-feedback） --- # HH-RLHF 数据集卡片 ## 数据集概述 本仓库提供两类不同数据的访问权限： 1. 来自论文《Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback》（https://arxiv.org/abs/2204.05862）的关于助手有用性与无害性的人类偏好数据。此类数据旨在训练偏好模型（或奖励模型）以用于后续的基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF）训练。此类数据**不可**用于对话智能体的监督训练，在该类数据上训练对话智能体可能会生成有害模型，需避免此类操作。 2. 来自论文《Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned》（https://www.anthropic.com/red_teaming.pdf）的人类生成并标注的红队测试对话数据。此类数据用于探究众包工作者如何对模型开展红队测试，以及哪些类型的红队攻击能够成功或无法成功。此类数据**不可**用于微调或偏好建模（偏好建模请使用前述第一类数据）。该类数据均为对话完整转录文本，源自上文提及的无害性偏好建模数据，且仅将被选中的回复纳入整体转录内容。此外，转录文本已通过人工与自动化手段标注了整体对话的有害程度。 **免责声明**：本数据集（尤其是无害性偏好数据与红队测试数据）可能包含冒犯性或令人不适的内容，涉及但不限于歧视性言论、虐待、暴力、自残、剥削及其他可能引发不适的主题。请仅根据自身风险承受能力处理此类数据。本数据集仅用于研究目的，尤其是旨在降低模型有害性的研究。数据中表达的观点不代表Anthropic及其任何雇员的立场。如前文所述，此类数据**不可**用于训练对话智能体，否则可能导致模型产生有害行为。 下文将分别详述这两类数据集。 ## 关于有用性与无害性的人类偏好数据（偏好建模数据，PM Data） 此类数据的详细说明参见论文《Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback》（https://arxiv.org/abs/2204.05862）。若您认为本数据对研究有所助益，请引用该论文。数据格式极为简洁——jsonl文件的每一行均包含一对文本，分别为「选中回复（chosen）」与「拒绝回复（rejected）」。 在**有用性**维度，数据分为三个批次的训练/测试集：来自基础模型（上下文蒸馏的520亿参数语言模型）、基于早期偏好模型的拒绝采样（多数采用16选1采样）、以及在迭代「在线」流程中采样得到的数据集。 在**无害性**维度，数据仅针对基础模型采集，但格式与前述一致。 关于数据采集流程与众包工作者群体的详细信息，请参见论文的第2节与附录D。 ## 红队测试数据（非偏好建模数据） 此类数据的详细说明参见论文《Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned》（https://www.anthropic.com/red_teaming.pdf）。若您认为本数据对研究有所助益，请引用该论文。总体而言，通过分析此类数据可了解成功的红队测试尝试具有何种特征。 关于数据与采集流程的详细信息，请参见论文附录中的数据集说明文档（Datasheet）。 jsonl文件的每一行均为包含以下字段的字典： - `transcript`：人类攻击者（红队成员）与AI助手之间的对话文本转录内容 - `min_harmlessness_score_transcript`：从偏好模型得到的AI助手无害性实值评分（分值越低，有害性越高） - `num_params`：支撑该AI助手的语言模型参数量 - `model_type`：支撑该AI助手的模型类型 - `rating`：红队成员对自身成功突破AI助手的评分（李克特量表，分值越高代表突破越成功） - `task_description`：红队成员撰写的简短描述，说明其尝试对AI助手开展红队测试的方式 - `task_description_harmlessness_score`：从偏好模型得到的任务描述无害性实值评分（分值越低，有害性越高） - `red_team_member_id`：红队成员的任意标识符，一名红队成员可发起多次红队测试攻击 - `is_upworker`：二元标识变量，若红队成员来自众包平台Upwork则为真，若来自MTurk则为假 - `tags`：每条转录文本最多包含6个标签的列表。标签为事后审阅红队数据的众包工作者生成的红队测试尝试简短描述。仅针对四种模型类型中的两种，随机抽取的1000次红队测试攻击提供了标签。 ## 使用方法 上述两类数据集分别位于独立的子目录中。若需加载特定子集，请通过`load_dataset()`函数的`data_dir`参数实现，示例如下： python from datasets import load_dataset # 加载所有有用性/无害性子集（schema一致） dataset = load_dataset("Anthropic/hh-rlhf") # 加载单个无害性子集 dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base") # 加载红队测试子集 dataset = load_dataset("Anthropic/hh-rlhf", data_dir="red-team-attempts") ## 联系方式 数据集原作者将本数据集托管于GitHub：https://github.com/anthropics/hh-rlhf。您可通过邮箱 [e-mail] 提交咨询。