text-2-video-human-preferences-sora-2-pro

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Video Generation Sora 2 Pro Human Preference

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 Pro video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark).

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

# Overview

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 Pro video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.
The benchmark data is accessible on [huggingface](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences) directly.

# Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found [here](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences-moonvalley-marey/tree/main/Videos)

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

# Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

## Examples

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>A bustling city marketplace from dawn to dusk, capturing vendors setting up, colorful goods being sold, diverse crowds interacting, lights illuminating as night falls, and the vibrant energy transitioning between day and night.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_0_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Ray 2 </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/ray2_0000_2.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>A serene mermaid glides through vibrant coral reefs, sunlight filtering through the water. Fish of every color swim around her gracefully as she dances with the ocean currents.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 9.47%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_43_0.gif " width=500>
</div>
<div>
<h3 class="score-amount">Veo 2 </h3>
<div class="score-percentage">(Score: 90.53%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo2_0043_0.gif" width=500>
</div>
</div>
</div>
</div>

# Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Glitch Rating: 0%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_75_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Alpha </h3>
<div class="score-percentage">(Glitch Rating: 100%)</div>
<img src="https://assets.rapidata.ai/alpha_0075_359978639.gif " width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Glitch Rating: 88.95%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Glitch Rating: 11.05%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey-11-8-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

# Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 94.11%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_39_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Pika 2.2 </h3>
<div class="score-percentage">(Score: 5.89%)</div>
<img src="https://assets.rapidata.ai/pika2.2_0039_1.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_64_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Veo 3 </h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo3_0064_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

</br>

# About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development.

# Other Datasets

We run a benchmark of the major video generation models, the results can be found on our [website](https://www.rapidata.ai/leaderboard/video-models). We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

- Link to the [Rich Video Annotation dataset](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- Link to the [Coherence dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- Link to the [Text-2-Image Alignment dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- Link to the [Preference dataset](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Sora 2 Pro视频生成人类偏好数据集

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

本数据集共收集约1.5万名人类标注者的7.5万条有效反馈，用于在我们的基准测试中评估Sora 2 Pro视频生成模型。本数据集通过[Rapidata Python应用程序接口（API）](https://docs.rapidata.ai)耗时约30分钟完成采集，该接口面向所有用户开放，是大规模数据标注的理想工具。

您可访问我们的[官方网站](https://www.rapidata.ai/benchmark)查看最新的模型排名。

若本数据集对您有所助益，并希望未来看到更多相关资源，欢迎为其点赞❤️

## 概览
本数据集共收集约1.5万名人类标注者的7.5万条有效反馈，用于在我们的基准测试中评估Sora 2 Pro视频生成模型。本数据集通过[Rapidata Python应用程序接口（API）](https://docs.rapidata.ai)耗时约30分钟完成采集，该接口面向所有用户开放，是大规模数据标注的理想工具。
基准测试数据可直接在[Hugging Face平台](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences)获取。

## 列说明
本数据集包含成对视频对比样本。每条数据均包含`video1`与`video2`字段，其中存储了用于快速预览的压缩GIF格式视频链接。全分辨率视频可在此处[获取](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences-moonvalley-marey/tree/main/Videos)。
`weighted_results`列包含0至1区间的分数，代表汇总后的用户反馈结果。单条用户反馈详情可在`detailedResults`列中查看。

## 对齐度
对齐分数用于量化视频与对应提示词的匹配程度。标注者需回答：「哪段视频更贴合给定的描述？」

### 示例
<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>从黎明到黄昏的繁华城市市集：涵盖摊主筹备摊位、售卖各色商品、不同人群互动、夜幕降临灯光亮起，以及昼夜交替间的蓬勃活力</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(得分：100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_0_0.gif" width="500">
</div>
<div>
<h3 class="score-amount">Ray 2 </h3>
<div class="score-percentage">(得分：0%)</div>
<img src="https://assets.rapidata.ai/ray2_0000_2.gif" width="500">
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>悠然的美人鱼穿梭于色彩斑斓的珊瑚礁间，阳光穿透水面洒落。各色鱼类优雅地环绕她游动，她随洋流翩翩起舞</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(得分：9.47%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_43_0.gif" width="500">
</div>
<div>
<h3 class="score-amount">Veo 2 </h3>
<div class="score-percentage">(得分：90.53%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo2_0043_0.gif" width="500">
</div>
</div>
</div>
</div>

## 连贯性
连贯性分数用于评估生成视频是否符合逻辑，且无伪影或视觉瑕疵。在不查看原始提示词的前提下，标注者需回答：「哪段视频存在更多瑕疵，更像是AI生成内容？」

### 示例
<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(瑕疵率：0%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_75_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Alpha </h3>
<div class="score-percentage">(瑕疵率：100%)</div>
<img src="https://assets.rapidata.ai/alpha_0075_359978639.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(瑕疵率：88.95%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(瑕疵率：11.05%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey-11-8-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

## 偏好度
偏好度分数用于反映参与者对单段视频的视觉吸引力评价，与原始提示词无关。标注者需回答：「从美学角度出发，你更偏好哪段视频？」

### 示例
<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(得分：94.11%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_39_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Pika 2.2 </h3>
<div class="score-percentage">(得分：5.89%)</div>
<img src="https://assets.rapidata.ai/pika2.2_0039_1.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro </h3>
<div class="score-percentage">(得分：0%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_64_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Veo 3 </h3>
<div class="score-percentage">(得分：100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo3_0064_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

</br>

## 关于Rapidata
Rapidata的技术让大规模人类反馈采集比以往任何时候都更快捷、更易用。访问[官方网站](https://www.rapidata.ai/)，了解我们如何革新AI开发中的人类反馈采集流程。

## 其他数据集
我们对主流视频生成模型开展基准测试，测试结果可在[官方网站](https://www.rapidata.ai/leaderboard/video-models)查看。我们依据模型的连贯性/合理性、与提示词的对齐度以及风格偏好度进行排名。超过200万条的底层标注数据可通过以下链接获取：

- [富视频标注数据集](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- [连贯性数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- [文本到图像对齐数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- [偏好数据集](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)