image-to-video-human-preference-hailuo-02-marey

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
display: flex;
align-items:center;
flex-direction: column;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Video Generation Hailuo-02 v Marey Human Preference

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark).

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

# Overview

In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

# Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found [here](https://huggingface.co/datasets/Rapidata/image-to-video-human-preference-hailuo-02-marey/tree/main/Videos)

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

# Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

## Examples

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>A gentle breeze rustles the leaves and sways the grape cluster softly.</q>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/_YQZcHv13mIghrsjwtZtY.jpeg" width=300>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Score: 75.34%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0059.gif" width=500>
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Score: 24.66%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_scene-motion_0059.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>The camera gently circles the blooming rose, capturing its petals in soft focus.</q>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/8YP7x40b3h4BlIqUaGxg-.jpeg" width=300>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Score: 14.42%)</div>
<img src="https://assets.rapidata.ai/marey_0063.gif" width=500>
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Score: 85.58%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_camera-motion_0063.gif" width=500>
</div>
</div>
</div>
</div>

# Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Glitch Rating: 13.68%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0084.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Glitch Rating: 86.32%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_scene-motion_0084.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Glitch Rating: 88.92%)</div>
<img src="https://assets.rapidata.ai/marey_0003.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Glitch Rating: 11.08%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_style_0003.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

# Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Score: 78.21%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0009.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Score: 21.79%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_camera-motion_0009.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Score: 11.02%)</div>
<img src="https://assets.rapidata.ai/marey_0020.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02 </h3>
<div class="score-percentage">(Score: 88.98%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_scene-motion_0020.gif " width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

</br>

# About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development.

# Other Datasets

We run a benchmark of the major video generation models, the results can be found on our [website](https://www.rapidata.ai/leaderboard/video-models). We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

- Link to the [Rich Video Annotation dataset](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- Link to the [Coherence dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- Link to the [Text-2-Image Alignment dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- Link to the [Preference dataset](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)

<style>.vertical-container {display: flex;flex-direction: column;gap: 60px;}.image-container img {height: 150px;margin:0;object-fit: contain;width: auto;}.image-container {display: flex;justify-content: space-around;align-items: center;}.container {width: 90%;margin: 0 auto;}.text-center {text-align: center;display: flex;align-items:center;flex-direction: column;}.score-amount {margin: 0;margin-top: 10px;}.score-percentage {font-size: 12px;font-weight: semi-bold;}</style>

# Rapidata视频生成Hailuo-02与Marey人类偏好数据集

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

本数据集共收集了约2000名人类标注者提供的约6000条反馈，用于在我们的基准测试中评估Seedance 1 Pro视频生成模型。本数据集仅需约5分钟即可通过[Rapidata Python API](https://docs.rapidata.ai)完成采集，该工具面向所有用户开放，是大规模数据标注的理想方案。

可前往我们的[官方网站](https://www.rapidata.ai/benchmark)查看最新的模型排名榜单。

若您从本数据集获益并希望未来获取更多同类资源，欢迎为其点赞 ❤️

# 数据集概览

本数据集共收集了约2000名人类标注者提供的约6000条反馈，用于在我们的基准测试中评估Seedance 1 Pro视频生成模型。本数据集仅需约5分钟即可通过[Rapidata Python API](https://docs.rapidata.ai)完成采集，该工具面向所有用户开放，是大规模数据标注的理想方案。

# 列字段说明

本数据集包含成对的视频对比样本。每条数据均包含`video1`与`video2`字段，其中存储了用于快速预览的压缩版GIF链接。高清分辨率的完整视频可通过[此处](https://huggingface.co/datasets/Rapidata/image-to-video-human-preference-hailuo-02-marey/tree/main/Videos)获取。

`weighted_results`列存储了取值范围为0至1的分数，代表汇总后的用户反馈结果。单条用户的原始反馈可在`detailedResults`列中查看。

# 对齐度评分

对齐度评分用于量化视频与对应提示词的匹配程度。标注者需回答：“哪一段视频更贴合给定的描述？”

## 示例

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>微风轻拂，树叶沙沙作响，葡萄串轻柔摇曳。</q>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/_YQZcHv13mIghrsjwtZtY.jpeg" width=300>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(得分：75.34%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0059.gif" width=500>
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(得分：24.66%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_scene-motion_0059.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>镜头环绕盛放的玫瑰缓缓移动，以柔焦效果捕捉花瓣细节。</q>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/8YP7x40b3h4BlIqUaGxg-.jpeg" width=300>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(得分：14.42%)</div>
<img src="https://assets.rapidata.ai/marey_0063.gif" width=500>
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(得分：85.58%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_camera-motion_0063.gif" width=500>
</div>
</div>
</div>
</div>

# 连贯性评分

连贯性评分用于评估生成视频的逻辑自洽性，以及是否存在伪影或视觉瑕疵。标注者在看不到原始提示词的前提下需回答：“哪一段视频的瑕疵更多，且更像是AI生成的内容？”

## 示例

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(瑕疵评分：13.68%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0084.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(瑕疵评分：86.32%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_scene-motion_0084.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(瑕疵评分：88.92%)</div>
<img src="https://assets.rapidata.ai/marey_0003.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(瑕疵评分：11.08%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_style_0003.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

# 视觉偏好评分

视觉偏好评分用于反映参与者对单段视频的视觉美观度评价，不受提示词影响。标注者需回答：“从美学角度而言，你更偏好哪一段视频？”

## 示例

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(得分：78.21%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey_0009.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(得分：21.79%)</div>
<img src="https://assets.rapidata.ai/hailuo-02_camera-motion_0009.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Marey模型</h3>
<div class="score-percentage">(得分：11.02%)</div>
<img src="https://assets.rapidata.ai/marey_0020.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Hailuo-02模型</h3>
<div class="score-percentage">(得分：88.98%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_scene-motion_0020.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

<br/>

# 关于Rapidata

Rapidata的技术让大规模人类反馈采集变得比以往任何时候都更快捷、更易用。请访问[rapidata.ai](https://www.rapidata.ai/)了解更多关于我们如何革新AI开发领域的人类反馈采集技术的信息。

# 其他数据集

我们针对主流视频生成模型开展了基准测试，测试结果可在我们的[官方网站](https://www.rapidata.ai/leaderboard/video-models)查看。我们将根据模型的连贯性/合理性、与提示词的对齐度以及风格偏好性进行排名。背后支撑该榜单的200万+条标注数据可通过以下链接获取：

- [Rich Video Annotation数据集](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- [连贯性数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- [文本-图像对齐数据集](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- [偏好数据集](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)

搜集汇总

背景与挑战

背景概述

该数据集用于评估Seedance 1 Pro视频生成模型，包含约6k条来自2k名标注者的人类偏好响应，通过Rapidata Python API快速收集。它提供了视频对比数据，涵盖对齐度、连贯性和视觉偏好三个维度的评分。