JeremyAlain/SLF5K

---
annotations_creators:
- expert-generated
language:
- en
language_creators:
- found
license: apache-2.0
multilinguality:
- monolingual
pretty_name: SLF5K
size_categories:
- 1K<n<10K
source_datasets:
- original
tags:
- feedback
- human feedback
- language feedback
- binary feedback
- reward
- reward model
- gpt3
- gpt-3
- instructgpt
- alignment
- ai alignment
- scale
- imitation learning from language feedback
- ilf
task_categories:
- summarization
task_ids: []
---
# Dataset Card for SLF5K

## Dataset Description

- **Repository: https://github.com/JeremyAlain/imitation_learning_from_language_feedback**
- **Paper: Training Language Models with Language Feedback at Scale**
- **Point of Contact: [e-mail] and [e-mail]**

### Dataset Summary

The Summarization with Language Feedback (SLF5K) dataset is an English-language dataset containing 5K unique samples that can be used
for the task of abstraction summarization. Each sample consists
of a Reddit title and post, a model-generated ([FeedME](https://beta.openai.com/docs/model-index-for-researchers)) summary, and human-written language feedback on that summary.
Additionally, each sample has a high-quality, human-written (gold) summary that should be ideal for the Reddit post.
Lastly, each sample has two additional model-generated summaries with binary human preference labels, on which summary is preferred by a human.
The dataset can be used to train language models with language feedback on abstractive summarization. It can also be
used to train a reward model on binary preferences.

The Reddit posts were taken from the datasets provided by [Learning to Summarize from Human Feedbback](https://arxiv.org/pdf/2009.01325.pdf), who used the initial Reddit post dataset
[TL;DR: Mining Reddit to Learn Automatic Summarization](https://aclanthology.org/W17-4508.pdf).

### Supported Tasks and Leaderboards

The dataset can be used to train a model for abstractive and extractive summarization. It can either be trained directly on
human-written summaries, or leverage language feedback or binary human preferences.
The model performance is evaluated in a human evaluation, where annotators rate the quality of the generated summaries.
Previous work has used [ROUGE](https://huggingface.co/spaces/evaluate-metric/rouge) scores, but in [Learning to Summarize from Human Feedbback](https://arxiv.org/pdf/2009.01325.pdf) they
show that ROUGE is not an ideal metric.

### Languages

English

## Dataset Structure

### Data Instances

Each instance is a line in the dataset file (which is saved as .jsonl). Each instance contains various fields, where the most important are
Here is an example instance:
```
{"id":"t3_3w7gyp",
"subreddit":"dogs",
"title":"Puppy playing at park - other owner aggressive towards him [help]",
"post":"Hi all, looking for some advice. I have a 6m old kelpie, buzz, who goes with me daily to a dog park, [...]",
"tldr_human_reference_summary":"other owner at park harsh with my dog for playing to rough with his. Have tried talking to him about it, hasn't helped.",
"summary_prompt":"Write an excellent summary of the given text.\n\nTitle: Puppy playing at park - other owner aggressive towards him [help]\n\nText: Hi all, looking for some advice. [...] that too.\n\nTL;DR:",
"generated_summary_for_comparison_A":"New dog at park is being aggressive to my pup, owner won't stop. What do I do?",
"generated_summary_for_comparison_B":"A new dog has been coming to the dog park and the first day the new dog came, the old dog (a kelpie) was all over him.",
"generated_summary_for_feedback":"A new dog has been coming to the dog park and the first day the owner hauled buzz off and whacked him. Today, the owner was staring daggers at me and lunging at buzz\/pulling his collar roughly.",
"comparison_preference":"Summary A",
"feedback":"The summary is concise but could include information about the poster knowing the dogs are just playing and will react if they become aggressive and wants to know how to handle things with Max's dad. ",
"feedback_class":"Coverage",
"has_additional_feedback":"No",
"ideal_human_summary":"The poster is frustrated with a new person at the dog park who is upset with him because their young dogs are playing roughly. The poster will step in if it gets aggressive and wants the new person to understand this. "}
```

There are some additional fields like `time_spent_in_seconds_ideal_human_summary`, `time_spent_in_seconds_feedback`,`time_spent_in_seconds_comparison` which only have values for the development dataset.

### Data Fields

- `id`: a unique string identifying the reddit post.
- `subreddit`: subreddit of the post.
- `title`: title of the reddit post.
- `post`: reddit post
- `tldr_human_reference_summary`: human reference summary automatically extracted from reddit (taken from the dataset of [TL;DR: Mining Reddit to Learn Automatic Summarization](https://aclanthology.org/W17-4508.pdf))
- `summary_prompt`: the whole prompt used to generate summaries
- `generated_summary_for_comparison_A`: summary A used for binary human comparison (generated with FeedME)
- `generated_summary_for_comparison_B`: summary B used for binary human comparison (generated with FeedME)
- `generated_summary_for_feedback`: summary used to gather human language feedback ((generated with FeedME))
- `comparison_preference`: prefered Summary of human comparison, Values: "Summary A", "Summary B"
- `feedback`: human language feedback on `generated_summary_for_feedback`(most important feedback point)
- `feedback_class`: Class of language feedback, Values: "Coverage", "Accuracy", "Coherence", "other"
- `has_additional_feedback`: Whether this sample could use more feedback on an important point.
- `ideal_human_summary`: high-quality human-written summary for this sample. We instructed annotators to write an ideal summary.
- `time_spent_in_seconds_ideal_human_summary`: Annotation time for ideal human summary
- `time_spent_in_seconds_feedback`: Annotation time for language feedback
- `time_spent_in_seconds_comparison`: Annotation time for binary comparison

Note that the various datasplits have varying fields. The fields that are not contained in a dataset have the value None.

### Data Splits

The SLF5K dataset has 4 splits: _train_, _development_, _validation_, and _test_. Below are the statistics of the dataset.

| Dataset Split | Number of Instances in Split |
| ------------- | ------------------------------------------- |
| Train | 5000 |
| Development | 200 |
| Validation | 500 |
| Test | 698 |

The reason we introduce a development and validation dataset, is the following.

## Dataset Creation

### Curation Rationale

This dataset aims to support supervised language model training from human preferences on a summarization task with real natural training data.

### Source Data

#### Initial Data Collection and Normalization

The initial TL;DR dataset was made public by Völkse et. al. in the paper [TL;DR: Mining Reddit to Learn Automatic Summarization](https://aclanthology.org/W17-4508.pdf) (licensed under CC By 4.0).
Stiennon et. al. then use this TL;DR dataset for their work [Learning to Summarize from Human Feedbback](https://arxiv.org/pdf/2009.01325.pdf).
They filter the TL;DR dataset for quality reasons and collect binary human preference labels.

Our datset is a subset from Stiennon et. al. Dataset, which can be downloaded [here](https://github.com/openai/summarize-from-feedback).
Our train and development dataset are taken form their train dataset and our test and validation datasets are taken from their test datasest.

#### Who are the source language producers?

The reddit posts are written by users of reddit.com.

### Annotations

#### Annotation process

We first onboarded annotators by giving them test tasks on which we evaluated their annotation quality. We then selected 31
annotators for the remainder of the project (a few were removed later on due to quality issues). Througout the process
we updated our instructions to make the tasks clearer and stayed in close contact with the annotators to answer questions etc.

The various dataset splits were collected in multiple annotation iterations. The largest annotation was a single iteration of annotation
5000 samples for the train dataset.

#### Who are the annotators?

We used annotators through the annotation service [Surge AI](https://www.surgehq.ai/).

### Personal and Sensitive Information

The annotators were completely anonymized and no information about them can be found in the dataset.

## Considerations for Using the Data

### Social Impact of Dataset

The purpose of this dataset is to align language models with human preferences by leveraging language feedback, on the task of summarization. Concretely, the goal is to
to develop models that produce summaries for reddit posts that are more in line with human preferences.
Note that this does not imply that the outputs will perfectly be aligned with human values, i.e. outputs can still be misaligned, offensive and contain harumful biases.

While outputs from a model trained on our dataset may reflect the language of the reddit posts, summaries, and human feedback, it should always be made clear that such an output
is automatically generated.

### Discussion of Biases

The TL;DR dataset consists of user-submitted posts to the website reddit.com. It can thus contain content that is offensive or reflects harmful social biases.
We thus recommend that models trained on the SLF5K dataset (which is based on the TL;DR) dataset be thoroughly studied for potential harmful behavior.

The human preferences and feedback represented in this dataset were collected through crowd-workers and may disproportionally represent the views, biases, and values
of the respective demographic of the annotators.

### Other Known Limitations

The "human-summaries" collected in the TL;DR dataset (and available in the SLF5K dataset under the field `tldr_human_reference_summary`, were automatically extracted from reddit.com.
They are often of poor quality and do not accurately reflect human summarization performance. In our paper, we show that our human written summaries (available in the SLF5K dataset under the field
`ideal_human_summary`) are of much higher quality.

## Additional Information

### Dataset Curators

The data is collected by Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez.
All authors are affiliated with New York University. Additionally, Jérémy Scheurer is affiliated with FAR AI. Jon Ander
is affiliated with the University of the Basque Country. Tomek Korbak is affiliated with FAR AI and the University of Sussesx.
Kyunghyun Cho is affiliated with Genentech and CIFAR LMB. Ethan Perez is affiliated with FAR AI and Anthropic.

### Licensing Information

The SLF5K dataset is released under the Apache 2.0 license.

### Citation Information

TBD

原始信息汇总

数据集概述

数据集描述

• 名称: SLF5K
• 语言: 英语
• 数据来源: 原始数据，源自TL;DR: Mining Reddit to Learn Automatic Summarization和Learning to Summarize from Human Feedback
• 许可: Apache-2.0
• 数据集大小: 包含5000个样本
• 任务类型: 抽象摘要
• 数据集内容: 每个样本包含一个Reddit标题和帖子、一个模型生成的摘要、人类编写的语言反馈、高质量的人类编写（黄金）摘要以及两个额外的模型生成的摘要与二元人类偏好标签。

数据集结构

数据实例

• 格式: JSONL
• 关键字段:
  • id: 唯一标识Reddit帖子的字符串
  • subreddit: 帖子所属的子论坛
  • title: 帖子的标题
  • post: 帖子内容
  • tldr_human_reference_summary: 人类参考摘要
  • summary_prompt: 生成摘要的提示
  • generated_summary_for_comparison_A 和 generated_summary_for_comparison_B: 用于二元人类比较的模型生成摘要
  • generated_summary_for_feedback: 用于收集人类语言反馈的模型生成摘要
  • comparison_preference: 人类偏好的摘要
  • feedback: 人类语言反馈
  • feedback_class: 反馈类别
  • ideal_human_summary: 高质量的人类编写摘要

数据分割

• 分割类型: 训练、开发、验证和测试
• 实例数量:
  • 训练: 5000
  • 开发: 200
  • 验证: 500
  • 测试: 698

数据集创建

来源数据

• 原始数据: Reddit帖子
• 数据收集: 通过Surge AI的注释者进行
• 注释过程: 经过多轮注释迭代，最大一轮注释了5000个样本用于训练集

许可证和引用信息

• 许可证: Apache 2.0
• 引用信息: 待定