sora-video-generation-time-flow

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 250px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.prompt {
width: 100%;
text-align: center;
font-weight: bold;
font-size: 16px;
min-height: 60px;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
text-align: right;
}
.score-container {
display: flex;
gap: 1rem;
min-height: 30px;
}

.main-container {
display: flex;
flex-direction: row;
gap: 60px;
}

.good {
color: #18c54f;
}
.bad {
color: red;
}

</style>

# Rapidata Video Generation Time flow Annotation Dataset

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

<p>
If you get value from this dataset and would like to see more in the future, please consider liking it.
</p>

This dataset was collected in ~1 hour using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation.

# Overview

In this dataset, ~3700 human evaluators were asked to evaluate AI-generated videos based on how time flows in the video. The specific question posed was: "How does time pass in this video?"

# Calculation Details

## Weighted Ratio

The weighted ratio is calculated using the responses and their associated userScores. This metric provides insight into how annotator responses are distributed across different options and indicates how ambiguously a video might fit into various categories.

## Confidence

The confidence metric serves to binarize the results. While it considers all responses and their Scores like the weighted ratio, its purpose is to determine the most likely correct category based on response patterns rather than simple weighted averaging.

It was also used in the data collection, which continued until either reaching 30 responses per datapoint or achieving a confidence level of >0.999, whichever occurred first.

# Videos

The videos in the dataset viewer are previewed as scaled down gifs. The original videos are stored under [Files and versions](https://huggingface.co/datasets/Rapidata/sora-video-generation-alignment-likert-scoring/tree/main/Videos)

<h3>
How does time pass in this video?
</h3>

<div class="main-container">

<div class="container">
<div class="prompt">
<q>Floating up past window washers on skyscraper floors</q>
</div>
<div class="image-container">
<div>
<img src="https://assets.rapidata.ai/036_20250114_sora.gif" width=500>
<div class="score-container">
<div class="score-percentage good">confidence_normal: 0.5053</div>
<div class="score-percentage good">confidence_slow: 0.4947</div>
</div>
</div>
</div>
<div>The confidence score was split between "normal" and "slow" timing. While the static human figures suggest slow motion, the rope moves at normal speed, creating this ambiguity.</div>
</div>

<div class="container">
<div class="prompt">
<q>Rose blooming and spreading petals in time lapse</q>
</div>
<div class="image-container">
<div>
<img src="https://assets.rapidata.ai/070_20250114_sora.gif" width=500>
<div class="score-container">
<div class="score-percentage good">confidence_fast: 0.9993</div>
</div>
</div>
</div>
<div>This video demonstrates an interesting contrast: while the footage appears to be in slow motion, it actually shows a time-lapse sequence, leading annotators to correctly label it as "fast."</div>
</div>

</div>

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
height: 250px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
}

.container {
width: 90%;
margin: 0 auto;
}

.prompt {
width: 100%;
text-align: center;
font-weight: bold;
font-size: 16px;
min-height: 60px;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {
font-size: 12px;
font-weight: semi-bold;
text-align: right;
}
.score-container {
display: flex;
gap: 1rem;
min-height: 30px;
}

.main-container {
display: flex;
flex-direction: row;
gap: 60px;
}

.good {
color: #18c54f;
}
.bad {
color: red;
}

</style>

# Rapidata视频生成时序标注数据集

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="数据集可视化">
</a>

<a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback">
</a>

<p>若您从本数据集获益并希望未来获取更多同类资源，不妨为该数据集点赞。</p>

<p>本数据集通过<a href="https://docs.rapidata.ai">[Rapidata Python应用程序编程接口（Rapidata Python API）]</a>耗时约1小时完成采集，面向所有用户开放，非常适用于大规模数据标注任务。</p>

# 数据集概览

<p>在本数据集中，共邀请约3700名人类评估员基于视频的时序表现对AI生成视频进行评分，评估问题为：“该视频的时间流逝状态如何？”</p>

# 评分计算细则

## 加权比率

<p>加权比率基于标注者的回复及其对应的用户评分计算得出，该指标可反映标注者在不同选项间的回复分布情况，同时也能体现某一视频在不同类别归属上的模糊程度。</p>

## 置信度

<p>置信度指标用于将结果二值化。与加权比率类似，其同样会考虑所有回复及其对应的评分，但核心目标是基于回复模式而非简单的加权平均来确定最可能的正确类别。</p>

<p>数据采集阶段同样采用了该置信度指标，采集过程将持续至每个数据点获得30条标注回复，或置信度达到0.999以上时停止，以先满足的条件为准。</p>

# 视频说明

<p>数据集查看器中的视频均以缩小尺寸的GIF格式预览，原始视频文件存储于<a href="https://huggingface.co/datasets/Rapidata/sora-video-generation-alignment-likert-scoring/tree/main/Videos">[文件与版本]</a>路径下。</p>

<h3>该视频的时间流逝状态如何？</h3>

<div class="main-container">

<div class="container">
<div class="prompt">
<q>在摩天楼楼层间向上漂浮，途经擦窗工人</q>
</div>
<div class="image-container">
<div>
<img src="https://assets.rapidata.ai/036_20250114_sora.gif" width=500>
<div class="score-container">
<div class="score-percentage good">正常时序置信度：0.5053</div>
<div class="score-percentage good">慢速时序置信度：0.4947</div>
</div>
</div>
</div>
<div>该视频的置信度评分在“正常时序”与“慢速时序”间均分：静态的人形身影暗示慢动作，但绳索的运动速度却符合正常时序，由此产生了标注歧义。</div>
</div>

<div class="container">
<div class="prompt">
<q>玫瑰绽放并延时扩散花瓣</q>
</div>
<div class="image-container">
<div>
<img src="https://assets.rapidata.ai/070_20250114_sora.gif" width=500>
<div class="score-container">
<div class="score-percentage good">快速时序置信度：0.9993</div>
</div>
</div>
</div>
<div>该视频呈现出一处有趣的反差：画面看似为慢动作，但实际为延时摄影序列，因此标注者准确将其归类为“快速时序”。</div>
</div>

</div>