SafeSora

# Dataset Card for SafeSora

<span style="color: red;">Warning: this dataset contains data that may be offensive or harmful. The data are intended for research purposes, especially research that can make models less harmful. The views expressed in the data do not reflect the views of PKU-Alignment Team or any of its members. </span>

[[`🏠 Project Homepage`](https://sites.google.com/view/safe-sora)]
[[`🤗 SafeSora Datasets`](https://huggingface.co/datasets/PKU-Alignment/SafeSora)]
[[`🤗 SafeSora Label`](https://huggingface.co/datasets/PKU-Alignment/SafeSora-Label)]
[[`🤗 SafeSora Evaluation`](https://huggingface.co/datasets/PKU-Alignment/SafeSora-Eval)]
[[`⭐️ Github Repo`](https://github.com/PKU-Alignment/safe-sora)]

SafeSora is a human preference dataset designed to support safety alignment research in the text-to-video generation field, aiming to enhance the helpfulness and harmlessness of Large Vision Models (LVMs). It currently contains three types of data:

- A classification dataset (SafeSora-Label) of 57k+ Text-Video pairs, including multi-label classification of 12 harm labels for their text prompts and text-video pairs.
- A human preference dataset (SafeSora) of 51k+ instances in the text-to-video generation task, containing comparative relationships in terms of helpfulness and harmlessness, as well as four sub-dimensions of helpfulness.
- An evaluation dataset (SafeSora-Eval) containing 600 human-written prompts, with 300 being safety-neutral and another 300 constructed according to 12 harm categories as red-team prompts.

In the future, we will also open-source some baseline alignment algorithms that utilize these datasets.

# Human Preference Dataset

SafeSora is a human preference dataset of 51k+ instances in the text-to-video generation task, containing comparative relationships in terms of helpfulness and harmlessness, as well as four sub-dimensions of helpfulness.
Each data point comprising a user input and two generated videos. Through a heuristic-based annotation process, human preferences were obtained in terms of `helpfulness` or `harmlessness` dimensions.
Additionally, due to a pre-annotation heuristic process, human preferences on four helpfulness sub-dimensions were also included. These sub-dimensions are:

- `Instruction Following`
- `Correctness`
- `Informativeness`
- `Aesthetics`

The specific annotation process is as shown in the figure below:


# Visualization example of data points


# SafeSora 数据集卡片
<span style="color: red;">警告：本数据集包含可能具有冒犯性或危害性的数据，仅用于研究目的，尤其是旨在提升模型安全性的相关研究。数据中表达的观点不代表北京大学对齐团队（PKU-Alignment Team）及其任何成员的立场。</span>

[[`🏠 项目主页`](https://sites.google.com/view/safe-sora)]
[[`🤗 SafeSora 数据集`](https://huggingface.co/datasets/PKU-Alignment/SafeSora)]
[[`🤗 SafeSora 标签数据集`](https://huggingface.co/datasets/PKU-Alignment/SafeSora-Label)]
[[`🤗 SafeSora 评估数据集`](https://huggingface.co/datasets/PKU-Alignment/SafeSora-Eval)]
[[`⭐️ GitHub 仓库`](https://github.com/PKU-Alignment/safe-sora)]

SafeSora 是一款面向文本到视频生成领域安全对齐研究的人类偏好数据集，旨在提升大视觉模型（Large Vision Models, LVMs）的有用性与无害性。目前该数据集包含三类数据：
- 分类数据集（SafeSora-Label）：包含57k+条文本-视频对，支持针对其文本提示与文本-视频对的12种有害标签多分类任务。
- 文本到视频生成任务下的人类偏好数据集（SafeSora）：包含51k+条样本，涵盖有用性与无害性维度的对比关系，以及4项有用性子维度。
- 评估数据集（SafeSora-Eval）：包含600条人工撰写的提示词，其中300条为安全中性提示词，另外300条则依据12种有害类别构建为红队提示词（red-team prompts）。

后续我们还将开源基于上述数据集开发的基线对齐算法。

# 人类偏好数据集
SafeSora 是文本到视频生成任务下的人类偏好数据集，包含51k+条样本，涵盖有用性与无害性维度的对比关系，以及4项有用性子维度。
每条数据均包含1条用户输入与2个生成视频。通过基于启发式规则的标注流程，我们获取了人类在“有用性”或“无害性”维度上的偏好。
此外，得益于预标注启发式流程，数据集还包含人类在4项有用性子维度上的偏好，这些子维度分别为：
- 指令遵循（Instruction Following）
- 正确性（Correctness）
- 信息丰富度（Informativeness）
- 美学性（Aesthetics）

具体标注流程如下图所示：


# 数据样本可视化示例