Runway_Frames_t2i_human_preferences

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
max-height: 250px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
box-sizing: content-box;
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
gap: .5rem
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {Score:
font-size: 12px;
font-weight: semi-bold;
}

</style>

# Rapidata Frames Preference

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="400" alt="Dataset visualization">
</a>

This T2I dataset contains roughly 400k human responses from over 82k individual annotators, collected in just ~2 Days using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale evaluation.
Evaluating Frames across three categories: preference, coherence, and alignment.

Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark).

If you get value from this dataset and would like to see more in the future, please consider liking it.

## Overview

This T2I dataset contains roughly 400k human responses from over 82k individual annotators, collected in just ~2 Days.
Evaluating Frames across three categories: preference, coherence, and alignment.

The evaluation consists of 1v1 comparisons between Frames and seven other models: Aurora, Imagen-3, Flux-1.1-pro, Flux-1-pro, DALL-E 3, Midjourney-5.2, and Stable Diffusion 3.

## Data collection

Since Frames is not available through an API, the images were collected manually through the user interface. The date following each model name indicates when the images were generated.

## Alignment

The alignment score quantifies how well an video matches its prompt. Users were asked: "Which image matches the description better?".

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>A fluffy pillow and a leather belt</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/euaf3dNqbPX8FOv_NJyE3.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/d4mrGiQwyXUJvctRfeij8.jpeg" width=500>
</div>
</div>
</div>

<div class="container">
<div class="text-center">
<q>The bright green grass contrasted with the dull grey pavement.</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/BE61b_M6BFi4_p3LPXIMf.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/Uai3r4CYfjP5RWlvVsKwB.jpeg" width=500>
</div>
</div>
</div>
</div>

## Coherence

The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which image feels less weird or unnatural when you look closely? I.e., has fewer strange-looking visual errors or glitches?"

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/WyTd59dNx7Wn4se-iU62I.png" width=500>
</div>
<div>
<h3 class="score-amount">Aurora </h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/SaoJOjnEUdMR1RL37tEYR.png" width=500>
</div>
</div>
</div>

<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/XqKLU3n_oqOQ7DCq_XgvQ.png" width=500>
</div>
<div>
<h3 class="score-amount">Flux-1.1</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/IEw5mbZnXbtQKFIWIQkha.jpeg" width=500>
</div>
</div>
</div>
</div>

## Preference

The preference score reflects how visually appealing participants found each image, independent of the prompt. Users were asked: "Which image do you prefer?"

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/sSO8yJWye5r8olR3WuQiI.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/bQAGt4-Emz_MMLfGe-ydX.jpeg" width=500>
</div>
</div>
</div>

<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/2ScBTyn3M3gr1sjGIX8li.png" width=500>
</div>
<div>
<h3 class="score-amount">Midjourney </h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/LQQ26LDUnumKWJWfX_tLF.jpeg" width=500>
</div>
</div>
</div>
</div>

## About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development.

<style>

.vertical-container {
display: flex;
flex-direction: column;
gap: 60px;
}

.image-container img {
max-height: 250px; /* Set the desired height */
margin:0;
object-fit: contain; /* Ensures the aspect ratio is maintained */
width: auto; /* Adjust width automatically based on height */
box-sizing: content-box;
}

.image-container {
display: flex; /* Aligns images side by side */
justify-content: space-around; /* Space them evenly */
align-items: center; /* Align them vertically */
gap: .5rem
}

.container {
width: 90%;
margin: 0 auto;
}

.text-center {
text-align: center;
}

.score-amount {
margin: 0;
margin-top: 10px;
}

.score-percentage {Score:
font-size: 12px;
font-weight: semi-bold;
}

</style>

<a href="https://www.rapidata.ai"><img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="400" alt="数据集可视化"></a>

# Rapidata Frames 偏好数据集

本文本到图像（Text-to-Image, T2I）数据集共收集了来自8.2万余名标注者的约40万条人类反馈，仅用约2天时间便通过[Rapidata Python API](https://docs.rapidata.ai)完成采集，面向所有用户开放，是大规模模型评估的理想选择。

该数据集围绕偏好性、一致性与对齐性三大维度对Frames模型生成内容进行评估。

可前往我们的[官网](https://www.rapidata.ai/benchmark)查看最新的模型排名。

如果您从本数据集获益并希望未来获取更多同类资源，欢迎为其点赞。

## 概述

本T2I数据集共收集了来自8.2万余名标注者的约40万条人类反馈，采集周期仅约2天。数据集围绕偏好性、一致性与对齐性三大维度对Frames模型进行评估。

本次评估采用1v1对比形式，将Frames与其余7款模型进行比对，分别为：Aurora、Imagen-3、Flux-1.1-pro、Flux-1-pro、DALL-E 3、Midjourney-5.2以及Stable Diffusion 3。

## 数据采集

由于Frames未开放API接口，相关图像均通过用户界面手动采集。每个模型名称后的标注日期代表该模型生成对应图像的时间。

## 对齐性

对齐性评分用于量化图像与文本提示的匹配程度。参与者被要求回答："哪张图像更贴合给定描述？"

<div class="vertical-container">
<div class="container">
<div class="text-center">
<q>蓬松的枕头与皮革腰带</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/euaf3dNqbPX8FOv_NJyE3.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/d4mrGiQwyXUJvctRfeij8.jpeg" width=500>
</div>
</div>
</div>

<div class="container">
<div class="text-center">
<q>鲜绿的草地与暗沉的灰色人行道形成鲜明对比</q>
</div>
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/BE61b_M6BFi4_p3LPXIMf.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/Uai3r4CYfjP5RWlvVsKwB.jpeg" width=500>
</div>
</div>
</div>
</div>

## 一致性

一致性评分用于衡量生成视频是否符合逻辑一致性，且无人工痕迹或视觉瑕疵。在不查看原始文本提示的前提下，参与者被要求回答："仔细观察后，哪张图像看起来更不怪异、更自然？换言之，哪张图像的视觉错误或故障更少？"

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/WyTd59dNx7Wn4se-iU62I.png" width=500>
</div>
<div>
<h3 class="score-amount">Aurora </h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/SaoJOjnEUdMR1RL37tEYR.png" width=500>
</div>
</div>
</div>

<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/XqKLU3n_oqOQ7DCq_XgvQ.png" width=500>
</div>
<div>
<h3 class="score-amount">Flux-1.1</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/IEw5mbZnXbtQKFIWIQkha.jpeg" width=500>
</div>
</div>
</div>
</div>

## 偏好性

偏好性评分用于体现参与者对每张图像的视觉喜爱程度，不受文本提示的影响。参与者被要求回答："你更偏好哪张图像？"

<div class="vertical-container">
<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/sSO8yJWye5r8olR3WuQiI.png" width=500>
</div>
<div>
<h3 class="score-amount">Imagen-3</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/bQAGt4-Emz_MMLfGe-ydX.jpeg" width=500>
</div>
</div>
</div>

<div class="container">
<div class="image-container">
<div>
<h3 class="score-amount">Frames</h3>
<div class="score-percentage">Score: 0%</div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/2ScBTyn3M3gr1sjGIX8li.png" width=500>
</div>
<div>
<h3 class="score-amount">Midjourney </h3>
<div class="score-percentage">Score: 100%</div>
<img style="border: 3px solid #18c54f;" src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/LQQ26LDUnumKWJWfX_tLF.jpeg" width=500>
</div>
</div>
</div>
</div>

## 关于Rapidata

Rapidata的技术让大规模人类反馈的采集速度更快、门槛更低，远超以往方案。欢迎访问[rapidata.ai](https://www.rapidata.ai/)，了解我们如何革新AI开发中的人类反馈采集流程。