Rapidata/text-2-video-human-preferences-sora-2-pro

---
dataset_info:
features:
- name: prompt
dtype: string
- name: video1
dtype: string
- name: video2
dtype: string
- name: weighted_results1_Alignment
dtype: float64
- name: weighted_results2_Alignment
dtype: float64
- name: detailedResults_Alignment
list:
- name: userDetails
struct:
- name: age
dtype: string
- name: country
dtype: string
- name: gender
dtype: string
- name: language
dtype: string
- name: occupation
dtype: string
- name: userScores
struct:
- name: global
dtype: float64
- name: votedFor
dtype: string
- name: weighted_results1_Coherence
dtype: float64
- name: weighted_results2_Coherence
dtype: float64
- name: detailedResults_Coherence
list:
- name: userDetails
struct:
- name: age
dtype: string
- name: country
dtype: string
- name: gender
dtype: string
- name: language
dtype: string
- name: occupation
dtype: string
- name: userScores
struct:
- name: global
dtype: float64
- name: votedFor
dtype: string
- name: weighted_results1_Preference
dtype: float64
- name: weighted_results2_Preference
dtype: float64
- name: detailedResults_Preference
list:
- name: userDetails
struct:
- name: age
dtype: string
- name: country
dtype: string
- name: gender
dtype: string
- name: language
dtype: string
- name: occupation
dtype: string
- name: userScores
struct:
- name: global
dtype: float64
- name: votedFor
dtype: string
- name: file_name1
dtype: string
- name: file_name2
dtype: string
- name: model1
dtype: string
- name: model2
dtype: string
splits:
- name: train
num_bytes: 6495323
num_examples: 1485
download_size: 632907
dataset_size: 6495323
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
license: apache-2.0
task_categories:
- video-classification
- text-to-video
- text-classification
language:
- en
tags:
- videos
- t2v
- text-2-video
- text2video
- text-to-video
- human
- annotations
- preferences
- likert
- coherence
- alignment
- wan
- wan 2.1
- veo2
- veo
- pikka
- alpha
- sora
- hunyuan
- veo3
- mochi-1
- seedance-1-pro
- seedance
- seedance 1
- Marey
- moonvalley
- sora2
- openai
- sora2 pro
pretty_name: Sora 2 Pro Human Preferences
size_categories:
- 1K<n<10K
---

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 150px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Video Generation Sora 2 Pro Human Preference

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 Pro video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark).

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

# Overview

In this dataset, ~75k human responses from ~15k human annotators were collected to evaluate the Sora 2 Pro video generation model on our benchmark. This dataset was collected in roughtly 30 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.
The benchmark data is accessible on [huggingface](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences) directly.

# Explanation of the colums

The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found [here](https://huggingface.co/datasets/Rapidata/text-2-video-human-preferences-moonvalley-marey/tree/main/Videos)

The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column.

# Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?".

## Examples

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>A bustling city marketplace from dawn to dusk, capturing vendors setting up, colorful goods being sold, diverse crowds interacting, lights illuminating as night falls, and the vibrant energy transitioning between day and night.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_0_0.gif" width=500>
</div>
<div>
<h3 class="score-amount">Ray 2 </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/ray2_0000_2.gif" width=500>
</div>
</div>
</div>
<div class="container">
<div class="text-center">
<q>A serene mermaid glides through vibrant coral reefs, sunlight filtering through the water. Fish of every color swim around her gracefully as she dances with the ocean currents.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 9.47%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_43_0.gif " width=500>
</div>
<div>
<h3 class="score-amount">Veo 2 </h3>
<div class="score-percentage">(Score: 90.53%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo2_0043_0.gif" width=500>
</div>
</div>
</div>
</div>

# Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Glitch Rating: 0%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_75_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Alpha </h3>
<div class="score-percentage">(Glitch Rating: 100%)</div>
<img src="https://assets.rapidata.ai/alpha_0075_359978639.gif " width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Glitch Rating: 88.95%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Marey </h3>
<div class="score-percentage">(Glitch Rating: 11.05%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/marey-11-8-25_24_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

# Preference

The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?"

## Examples

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro</h3>
<div class="score-percentage">(Score: 94.11%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/sora2-pro_9-10-25_39_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Pika 2.2 </h3>
<div class="score-percentage">(Score: 5.89%)</div>
<img src="https://assets.rapidata.ai/pika2.2_0039_1.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Sora2 Pro </h3>
<div class="score-percentage">(Score: 0%)</div>
<img src="https://assets.rapidata.ai/sora2-pro_9-10-25_64_0.gif" width="500" alt="Dataset visualization">
</div>
<div>
<h3 class="score-amount">Veo 3 </h3>
<div class="score-percentage">(Score: 100%)</div>
<img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/veo3_0064_0.gif" width="500" alt="Dataset visualization">
</div>
</div>
</div>
</div>

</br>

# About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development.

# Other Datasets

We run a benchmark of the major video generation models, the results can be found on our [website](https://www.rapidata.ai/leaderboard/video-models). We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here:

- Link to the [Rich Video Annotation dataset](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback)
- Link to the [Coherence dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset)
- Link to the [Text-2-Image Alignment dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset)
- Link to the [Preference dataset](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)