alvarobartt/HelpSteer-AIF

---
language:
- en
license: cc-by-4.0
size_categories:
- n<1K
pretty_name: HelpSteer with AIF
dataset_info:
features:
- name: prompt
dtype: string
- name: response
dtype: string
- name: model
dtype: string
- name: correctness
dtype: int64
- name: coherence
dtype: int64
- name: complexity
dtype: int64
- name: verbosity
dtype: int64
- name: helpfulness
dtype: int64
splits:
- name: train
num_bytes: 2832095
num_examples: 1000
download_size: 677100
dataset_size: 2832095
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
tags:
- synthetic
- distilabel
- helpsteer
- ai-feedback
- preference
---

<img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-dark.png" alt="Built with Distilabel" width="200" height="32"/>

# HelpSteer: Helpfulness SteerLM Dataset

HelpSteer is an open-source Helpfulness Dataset (CC-BY-4.0) that supports aligning models to become more helpful, factually correct and coherent, while being adjustable in terms of the complexity and verbosity of its responses.

[HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](http://arxiv.org/abs/2311.09528)

## Disclaimer

This is only a subset created with `distilabel` to evaluate the first 1000 rows using AI Feedback (AIF) coming from GPT-4, only created for experimenting / research purposes, please refer to [nvidia/HelpSteer](https://hf.co/nvidia/HelpSteer) if you want more information about the HelpSteer dataset.

## Dataset Description

HelpSteer contains 37120 samples, while this subset only contains the first 1000, each only containing a prompt and a response, even though the same prompt may appear up to 4 times with different responses generated by their in-house LLM of 43B params.

In this case, the annotations of the attributes have been discarded while just keeping the prompt and the response, to generate the annotations using AIF via `distilabel`.

## Attributes

1. **Helpfulness**: Overall helpfulness of the response to the prompt.
2. **Correctness**: Inclusion of all pertinent facts without errors.
3. **Coherence**: Consistency and clarity of expression.
4. **Complexity**: Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
5. **Verbosity**: Amount of detail included in the response, relative to what is asked for in the prompt.

## Source

1. (original) Prompts are collected based on a mixture of template-generated (mainly for prompt involving long reference text) and human generated by Scale AI. These prompts relate to the tasks of Rewrite, Summarization, Classification, Extraction, Closed Question Answering, Open Question Answering, Generation and Brainstorming.
2. (original) Responses are generated by an early version of an inhouse LLM. We generate up to 4 responses per prompts using sample techniques to give diverse yet reasonable responses.
3. (distilabel) Annotations of various attributes were done using OpenAI's GPT-4 via `distilabel`, following the same Likert 5 scale (0-4) that Scale AI used with human annotators, but this time asking GPT-4 to provide those, via AI Feedback (AIF).

## Citation

If you find this dataset useful, make sure to cite the original work, as the prompt and the responses have been reused from them, while only the annotations have been modified.

```bibtex
@misc{wang2023helpsteer,
title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM},
author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
year={2023},
eprint={2311.09528},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

原始信息汇总

HelpSteer: Helpfulness SteerLM Dataset

数据集概述

HelpSteer是一个开放源代码的有用性数据集（CC-BY-4.0），支持模型变得更加有用、事实正确和连贯，同时可以根据响应的复杂性和冗长性进行调整。

数据集描述

HelpSteer包含37120个样本，而此子集仅包含前1000个样本，每个样本仅包含一个提示和一个响应，尽管相同的提示可能出现多达4次，并生成不同的响应。

在此情况下，属性的注释已被丢弃，仅保留提示和响应，以便通过distilabel使用AI反馈（AIF）生成注释。

属性

1. Helpfulness：响应对提示的整体有用性。
2. Correctness：包含所有相关事实且无错误。
3. Coherence：表达的一致性和清晰性。
4. Complexity：编写响应所需的智力深度（即响应是否可以由具有基本语言能力的人编写，或者需要深入的领域专业知识）。
5. Verbosity：响应中包含的详细程度，相对于提示中要求的。

来源

1. （原始）提示基于模板生成（主要用于涉及长参考文本的提示）和Scale AI的人工生成。这些提示涉及重写、总结、分类、提取、封闭式问答、开放式问答、生成和头脑风暴等任务。
2. （原始）响应由早期版本的内部大型语言模型生成。我们使用采样技术为每个提示生成多达4个响应，以提供多样且合理的响应。
3. （distilabel）使用OpenAI的GPT-4通过distilabel对各种属性进行注释，遵循与Scale AI使用的人工注释相同的Likert 5级（0-4），但这次要求GPT-4提供这些注释，通过AI反馈（AIF）。

引用

如果您发现此数据集有用，请确保引用原始工作，因为提示和响应已从他们那里重新使用，而注释已修改。

bibtex @misc{wang2023helpsteer, title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM}, author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev}, year={2023}, eprint={2311.09528}, archivePrefix={arXiv}, primaryClass={cs.CL} }