HelpSteer

# HelpSteer: Helpfulness SteerLM Dataset

HelpSteer is an open-source Helpfulness Dataset (CC-BY-4.0) that supports aligning models to become more helpful, factually correct and coherent, while being adjustable in terms of the complexity and verbosity of its responses.

Leveraging this dataset and SteerLM, we train a Llama 2 70B to reach **7.54** on MT Bench, the highest among models trained on open-source datasets based on [MT Bench Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) as of 15 Nov 2023.
This model is available on HF at [Llama2-70B-SteerLM-Chat](https://huggingface.co/nvidia/Llama2-70B-SteerLM-Chat).

Try this model instantly for free hosted by us at [NVIDIA AI Playground](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/llama2-70b-steerlm). You can use this in the provided UI or through a limited access API (up to 10, 000 requests within 30 days).
If you would need more requests, we demonstrate how you can set up an inference server at [Llama2-70B-SteerLM-Chat model page on HF](https://huggingface.co/nvidia/Llama2-70B-SteerLM-Chat)

You can also train a model using [NeMo Aligner](https://github.com/NVIDIA/NeMo-Aligner) following [SteerLM training user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/modelalignment/steerlm.html).

<img src="https://huggingface.co/datasets/nvidia/HelpSteer/resolve/main/mtbench_categories.png" alt="MT Bench Categories" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

HelpSteer Paper : [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](http://arxiv.org/abs/2311.09528)

SteerLM Paper: [SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF](https://arxiv.org/abs/2310.05344)

Note: [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2) is the follow up to this dataset and we recommend using HelpSteer2 instead of this dataset.

## Dataset Description

HelpSteer contains 37, 120 samples, each containing a prompt, a response as well as five human-annotated attributes of the response, each ranging between 0 and 4 where higher means better for each attribute.

These attributes are:

1. **Helpfulness**: Overall helpfulness of the response to the prompt.
2. **Correctness**: Inclusion of all pertinent facts without errors.
3. **Coherence**: Consistency and clarity of expression.
4. **Complexity**: Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
5. **Verbosity**: Amount of detail included in the response, relative to what is asked for in the prompt.

## Example

```json
{
"prompt": "What are the three most important things to consider when deciding what technology to use to build an assist device to help an elderly person with basic needs?",
"response": "To build an assistive device to help an elderly person with basic needs, one must consider three crucial things: safety, compatibility, and ease of use. Safety is paramount, as the device must not cause harm to the user. Compatibility with the user's environment and other devices is also essential. Finally, the device must be simple enough for the elderly person to operate.",
"helpfulness": 3,
"correctness": 4,
"coherence": 4,
"complexity": 2,
"verbosity": 2
}
```

## Using the Huggingface Datasets

```python
from datasets import load_dataset

ds = load_dataset("nvidia/HelpSteer")

train = ds['train'] # len(train) = 35331 (95%)
val = ds['validation'] # len(val) = 1789 (5%)
```

## Source

1. Prompts are collected based on a mixture of template-generated (mainly for prompt involving long reference text) and human generated by Scale AI. These prompts relate to the tasks of Rewrite, Summarization, Classification, Extraction, Closed Question Answering, Open Question Answering, Generation and Brainstorming.
2. Responses are generated by an early version of an inhouse LLM. We generate up to 4 responses per prompts using sample techniques to give diverse yet reasonable responses.
3. Annotations of various attributes were done by Scale AI. Annotators rated each response on a Likert 5 scale (between 0 and 4) for each attribute (helpfulness, correctness, coherence, complexity and verbosity).

## Annotation methodology (short)

1. We engaged a select group of contractors via Scale AI. These contractors were provided with comprehensive guidelines that defined each attribute and the criteria for every rating level, together with some annotated examples. These guidelines and examples are detailed in the Appendix of the accompanying paper.
2. The annotation process involved approximately 200 U.S.-based human annotators. Candidates first underwent preliminary assignments, including assessments of English proficiency, to determine eligibility for working on the project. Subsequently, they participated in an introductory training course on the task which ended with a test that involved annotating 35 sample responses. This process ensured not only a thorough understanding of the task requirements but also the delivery of high-quality annotations.
3. Post-annotations, Scale AI performed extensive quality assurance, with each annotation reaching a minimum of two human reviews in addition to automated checks. After receiving the annotations from Scale AI, we conducted our independent quality assurance to make sure that the quality of the annotations was up to our expectations. As a result, some annotations were filtered away to retain only 37, 120 samples.

## Ethical statement
Annotators for the dataset were contracted through Scale AI. Scale AI engages the Anker Methodology, GISC Impact Sourcing Standard, and UN Sustainable Development Goals to provide a fair and competitive pay. The specific pay is calculated based on many factors, including the specific project, the specialized skillset and expertise required, regional costs of living and then transparently listed on Scale AI platform. Scale AI also provides multiple channels for questions and support, including 24/7 support teams, community discussion channels with specially trained moderators, and a “speak up” hotline where contractors can report concerns anonymously. Worker concerns can be submitted to and are reviewed by our Remotasks support team, and pay disputes are reviewed by support specialists trained in this area.

## Citation

If you find this dataset useful, please cite the following works

```bibtex
@misc{wang2023helpsteer,
title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM},
author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
year={2023},
eprint={2311.09528},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

```bibtex
@misc{dong2023steerlm,
title={SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF},
author={Yi Dong and Zhilin Wang and Makesh Narsimhan Sreedhar and Xianchao Wu and Oleksii Kuchaiev},
year={2023},
eprint={2310.05344},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

# HelpSteer：有益性SteerLM数据集

HelpSteer是一个遵循CC-BY-4.0协议的开源有益性数据集，旨在支持模型对齐，使模型输出更具有益性、事实准确性与连贯性，同时可根据响应的复杂度和详细程度进行调整。

借助本数据集与SteerLM，我们针对Llama 2 70B模型进行微调，使其在MT Bench评测中取得7.54分的成绩。截至2023年11月15日，该成绩在基于开源数据集训练的模型中位列[MT Bench排行榜](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)首位。该模型已在Hugging Face（HF）平台发布，地址为[Llama2-70B-SteerLM-Chat](https://huggingface.co/nvidia/Llama2-70B-SteerLM-Chat)。

你可通过我们托管的[NVIDIA AI Playground](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/llama2-70b-steerlm)免费即时体验该模型。你可通过内置UI或受限访问API进行调用（30天内最多可发起10000次请求）。若你需要更多请求额度，我们在HF平台的[Llama2-70B-SteerLM-Chat模型页面](https://huggingface.co/nvidia/Llama2-70B-SteerLM-Chat)中提供了部署推理服务器的教程。

你也可参考[SteerLM训练用户指南](https://docs.nvidia.com/nemo-framework/user-guide/latest/modelalignment/steerlm.html)，使用[NeMo Aligner](https://github.com/NVIDIA/NeMo-Aligner)进行模型训练。


HelpSteer论文：[HelpSteer：面向SteerLM的多属性有益性数据集](http://arxiv.org/abs/2311.09528)
SteerLM论文：[SteerLM：以属性条件化监督微调作为（用户可控）人类反馈强化学习（RLHF）的替代方案](https://arxiv.org/abs/2310.05344)

注意：[HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2)是本数据集的后续版本，我们推荐使用HelpSteer2而非本数据集。

## 数据集描述

HelpSteer共包含37120条样本，每条样本均包含一则提示词、一则模型响应，以及针对该响应的五项人工标注属性，所有属性的评分范围均为0至4分，分值越高代表表现越好。

标注属性如下：
1. **有益性（Helpfulness）**：响应对提示词的整体有益程度。
2. **准确性（Correctness）**：响应包含所有相关事实且无错误。
3. **连贯性（Coherence）**：表达的一致性与清晰度。
4. **复杂度（Complexity）**：撰写响应所需的智力深度（即该响应是仅具备基础语言能力的普通人即可完成，还是需要深厚的专业领域知识）。
5. **详细程度（Verbosity）**：相较于提示词的要求，响应中包含的细节量。

## 示例

json
{
"提示词": "在选择技术搭建辅助设备以帮助老年人满足基本生活需求时，需要考虑的三个最重要因素是什么？",
"模型响应": "要搭建帮助老年人满足基本生活需求的辅助设备，需考量三个关键要素：安全性、兼容性与易用性。安全性至关重要，设备不得对用户造成伤害。与用户使用环境及其他设备的兼容性也必不可少。最后，设备必须足够简单，便于老年人操作。",
"有益性": 3,
"准确性": 4,
"连贯性": 4,
"复杂度": 2,
"详细程度": 2
}

## 使用Hugging Face数据集工具

python
from datasets import load_dataset

# 加载HelpSteer数据集
ds = load_dataset("nvidia/HelpSteer")

# 训练集：共35331条样本，占比95%
train = ds['train']
# 验证集：共1789条样本，占比5%
val = ds['validation']

## 数据来源

1. 提示词来源：混合使用模板生成（主要用于涉及长参考文本的提示词）与Scale AI团队人工生成的提示词。这些提示词涵盖改写、摘要、分类、信息抽取、封闭式问答、开放式问答、内容生成与头脑风暴等任务类型。
2. 模型响应生成：由内部早期版本的大语言模型生成。我们通过采样技术为每条提示词生成最多4种响应，以保证响应的多样性与合理性。
3. 属性标注：由Scale AI团队完成。标注人员针对每条响应的五项属性（有益性、准确性、连贯性、复杂度与详细程度），采用李克特5级量表（0至4分）进行评分。

## 标注流程（精简版）

1. 标注团队招募：通过Scale AI平台招募经过筛选的签约标注人员。为标注人员提供了详尽的标注指南，明确了各项属性的定义、各评分等级的判定标准，并附带了标注示例。该指南与示例的详细内容可参见配套论文的附录部分。
2. 标注流程：共有约200名美国本土标注人员参与。候选标注人员首先需通过初步考核，包括英语能力测试，以确认是否具备项目参与资格。随后，他们将参加针对本任务的入门培训课程，课程结束后需完成一项包含35条样本响应的标注测试。该流程确保标注人员充分理解任务要求，从而输出高质量的标注结果。
3. 质量管控：标注完成后，Scale AI平台执行了严格的质量保障流程，每条标注除自动化检查外，至少需经过两名人工审核。我们在收到Scale AI提交的标注结果后，也开展了独立的质量校验工作，以确保标注质量符合预期。最终经过筛选，保留了37120条有效样本。

## 伦理声明

本数据集的标注人员均通过Scale AI平台签约。Scale AI平台采用Anker Methodology、GISC影响力采购标准（Impact Sourcing Standard）与联合国可持续发展目标，为标注人员提供公平且具有竞争力的薪酬。具体薪酬根据项目类型、所需专业技能与知识、地区生活成本等多因素计算，并在Scale AI平台上透明公示。此外，Scale AI还提供多种问题反馈与支持渠道，包括全天候客服团队、配备专业训练版主的社区讨论频道，以及匿名举报热线，签约人员可通过该热线匿名反馈相关问题。标注人员的诉求可提交至我们的Remotasks支持团队进行处理，薪酬纠纷则由经过专业培训的支持专员审核解决。

## 引用方式

若你认为本数据集对你的研究有所帮助，请引用以下论文：

bibtex
@misc{wang2023helpsteer,
title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM},
author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
year={2023},
eprint={2311.09528},
archivePrefix={arXiv},
primaryClass={cs.CL}
}

bibtex
@misc{dong2023steerlm,
title={SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF},
author={Yi Dong and Zhilin Wang and Makesh Narsimhan Sreedhar and Xianchao Wu and Oleksii Kuchaiev},
year={2023},
eprint={2310.05344},
archivePrefix={arXiv},
primaryClass={cs.CL}
}