text-2-video-human-preferences-sora-2

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Video Generation Sora 2 Human Preference

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark).

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

# Overview

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.
The benchmark data is accessible on [huggingface](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences) directly.

# Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found [here](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences-moonvalley-marey/tree/main/Videos)

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

# Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

## Examples

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>A split-screen showing different natural phenomena occurring simultaneously. Capture lightning striking mountains, waves crashing on cliffs, forests swaying in the wind, and clouds racing across the sky. Highlight the raw power, beauty, and harmony of nature's elements.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_87_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Alpha </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/alpha_0087_1286196315.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>A serene dawn view of a lone wolf traversing a misty forest, its fur illuminated by the early light as it moves gracefully through towering trees, embodying both solitude and strength.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_54_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Sora </h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora_0054_0.gif" width=500>
</div>
</div>
</div>
</div>

# Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Glitch Rating: 0%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_5_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hunyuan </h3>
<div class="score-percentage">(Glitch Rating: 100%)</div>
<img src="https://assets.rapidata.ai/hunyuan_0005_421.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Glitch Rating: 90.61%)</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_75_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Veo2 </h3>
<div class="score-percentage">(Glitch Rating: 9.39%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo2_0075_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

# Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Score: 95.44%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_4_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Mochi 1 </h3>
<div class="score-percentage">(Score: 4.56%)</div>
<img src="https://assets.rapidata.ai/mochi-1-24-7-25_4_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 </h3>
<div class="score-percentage">(Score: 10.16%)</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_64_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">seedance 1 pro </h3>
<div class="score-percentage">(Score: 89.84%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro-24-7-25_64_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

</br>

# About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development.

# Other Datasets

We run a benchmark of the major video generation models, the results can be found on our [website](https://www.rapidata.ai/leaderboard/video-models). We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

- Link to the [Rich Video Annotation dataset](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- Link to the [Coherence dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- Link to the [Text-2-Image Alignment dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- Link to the [Preference dataset](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata 视频生成 Sora 2 人类偏好数据集

<a href="https://www.rapidata.ai"><img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="数据集可视化"></a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback"></a>

本数据集共收集了约1.5万名人类标注者的约7.5万条标注结果，用于在我们的基准测试中评估Sora 2视频生成模型。本数据集可通过[Rapidata Python API](https://docs.rapidata.ai)在约30分钟内完成采集，该工具面向所有用户开放，是大规模数据标注的理想选择。

可访问我们的[官网](https://www.rapidata.ai/benchmark)查看最新的模型排名。

若您从本数据集获益并希望未来获取更多同类资源，欢迎点赞支持 ❤️

# 数据集概览
本数据集共收集了约1.5万名人类标注者的约7.5万条标注结果，用于在我们的基准测试中评估Sora 2视频生成模型。本数据集可通过[Rapidata Python API](https://docs.rapidata.ai)在约30分钟内完成采集，该工具面向所有用户开放，是大规模数据标注的理想选择。
基准测试数据可直接在[Hugging Face](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences)获取。

# 字段说明
本数据集包含成对的视频对比样本。每条数据均包含`video1`与`video2`字段，其中存储了用于快速预览的低分辨率GIF下载链接。完整分辨率的视频可通过[此处](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences-moonvalley-marey/tree/main/Videos)获取。
`weighted_results`字段包含取值范围为0至1的评分，代表聚合后的用户标注结果。单条用户标注的详细结果可在`detailedResults`字段中查看。

# 对齐性得分
对齐性得分用于量化视频与对应提示词的匹配程度。标注者被问及："哪段视频更贴合描述文本？"

## 示例
<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>分屏展示多种自然现象同步发生的场景：闪电击中山峦、海浪拍打悬崖、森林随风摇曳、云朵掠过天际，尽显自然元素的原始力量、美感与和谐统一。</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（得分：100%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_87_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Alpha</h3>
<div class="score-percentage">（得分：0%）</div>
<img src="https://assets.rapidata.ai/alpha_0087_1286196315.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>静谧的黎明时分，孤狼穿行于薄雾笼罩的森林，晨光轻抚其皮毛，它优雅地穿梭于参天林木之间，尽显孤傲与力量之态。</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（得分：0%）</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_54_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Sora</h3>
<div class="score-percentage">（得分：100%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora_0054_0.gif" width=500>
</div>
</div>
</div>
</div>

# 连贯性得分
连贯性得分用于衡量生成视频的逻辑自洽性，以及是否存在伪影或视觉瑕疵。标注者在不查看原始提示词的前提下被问及："哪段视频存在更多瑕疵，更有可能是AI生成内容？"

## 示例
<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（瑕疵评级：0%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_5_0.gif" width="500" alt="数据集可视化">
</div>
<div>
<h3 class="score-amount">Hunyuan</h3>
<div class="score-percentage">（瑕疵评级：100%）</div>
<img src="https://assets.rapidata.ai/hunyuan_0005_421.gif" width="500" alt="数据集可视化">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（瑕疵评级：90.61%）</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_75_0.gif" width="500" alt="数据集可视化">
</div>
<div>
<h3 class="score-amount">Veo2</h3>
<div class="score-percentage">（瑕疵评级：9.39%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo2_0075_0.gif" width="500" alt="数据集可视化">
</div>
</div>
</div>
</div>

# 审美偏好得分
审美偏好得分反映了参与者对视频视觉效果的喜爱程度，与提示词无关。标注者被问及："你更青睐哪段视频的美学表现？"

## 示例
<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（得分：95.44%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2_9-10-25_4_0.gif" width="500" alt="数据集可视化">
</div>
<div>
<h3 class="score-amount">Mochi 1</h3>
<div class="score-percentage">（得分：4.56%）</div>
<img src="https://assets.rapidata.ai/mochi-1-24-7-25_4_0.gif" width="500" alt="数据集可视化">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora 2</h3>
<div class="score-percentage">（得分：10.16%）</div>
<img src="https://assets.rapidata.ai/sora2_9-10-25_64_0.gif" width="500" alt="数据集可视化">
</div>
<div>
<h3 class="score-amount">seedance 1 pro</h3>
<div class="score-percentage">（得分：89.84%）</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro-24-7-25_64_0.gif" width="500" alt="数据集可视化">
</div>
</div>
</div>
</div>

<br>

# 关于Rapidata
Rapidata的技术让大规模人类反馈采集工作变得前所未有的高效便捷。访问[rapidata.ai](https://www.rapidata.ai/)了解更多我们如何革新AI开发中的人类反馈采集技术。

# 其他数据集
我们对主流视频生成模型开展了基准测试，测试结果可在[官网](https://www.rapidata.ai/leaderboard/video-models)查看。我们将根据模型的连贯性/合理性、与提示词的对齐程度以及风格偏好进行排名。相关的200余万条标注数据可通过以下链接获取：
- [丰富视频标注数据集](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- [连贯性数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- [文生图对齐数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- [偏好数据集](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)