open-image-preferences-v1-more-results-binarized

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="250" alt="Rapidata Logo">
</a>

We wanted to contribute to the challenge posed by the data-is-better-together community (description below). We collected 170'000 preferences using our API from people all around the world in rougly 3 days (docs.rapidata.ai):

If you get value from this dataset and would like to see more in the future, please consider liking it.

# Dataset Card for image-preferences-results [Original](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1)

<style>
.row {
display: flex;
justify-content: space-between;
width: 100%;
}
#container {
display: flex;
flex-direction: column;
font-family: Arial, sans-serif;
width: 98%
}
.prompt {
margin-bottom: 10px;
font-size: 16px;
line-height: 1.4;
color: #333;
background-color: #f8f8f8;
padding: 10px;
border-radius: 5px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
}
.image-container {
display: flex;
gap: 10px;
}
.column {
flex: 1;
position: relative;
}
img {
max-width: 100%;
height: auto;
display: block;
}
.image-label {
position: absolute;
top: 10px;
right: 10px;
background-color: rgba(255, 255, 255, 0.7);
color: black;
padding: 5px 10px;
border-radius: 5px;
font-weight: bold;
}
</style>

<div class="row">
<div class="column">
<div id="container">
<div class="prompt"><strong>Prompt:</strong> Anime-style concept art of a Mayan Quetzalcoatl biomutant, dystopian world, vibrant colors, 4K.</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1258.jpg">
<div class="image-label">Image 1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1258.jpg">
<div class="image-label">Image 2</div>
</div>
</div>
</div>
</div>
<div class="column">

<div id="container">
<div class="prompt"><strong>Prompt:</strong> 8-bit pixel art of a blue knight, green car, and glacier landscape in Norway, fantasy style, colorful and detailed.</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1210.jpg">
<div class="image-label">Image 1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1210.jpg">
<div class="image-label">Image 2</div>
</div>
</div>
</div>
</div>
</div>

- **Goal**: This project aims to create 10K text-to-image preference pairs. These pairs can be used to evaluate the performance of image generation models across a wide variety of common image categories, based on prompt with varying levels of difficulty.
- **How**: We use the prompts from [fal/imgsys-results](https://huggingface.co/datasets/fal/imgsys-results), these prompts are evolved based on complexity and quality for various image categories. We then asked the community to annotate the preference between two generated images for each prompt.
- **Result**: Rapidata collected over 170'000 individual preferences from people all around the globe. There were 17k image pairs for each of them we collected roughly 10 preference annotations.
- **Methodology**: Annotators were asked "Which image do you prefer based on the description?". They were given the option to choose between the two images, the prompt was also displayed. "Both" was not given as an option. Each pair was shown to 10 annoatators, the positions of the images were shuffled at random. Each Annotator has a trust worthyness score attached to each annotation, which can be found in the detailed results in the dataset.
- **Format**: In this version the results were binarized. For each pair we calcualted the user-score weighted rating between the two images and list a choosen and rejected model. Note that we have discarded all pairs where there was not a somewhat clear favorite, which we defined as a minimum 0.6 user-score weighted ratio for one of the images. [Our original dataset](https://huggingface.co/datasets/Rapidata/open-image-preferences-v1-more-results) has all the underlying information if you want to generate you own binarized version with your own threshold.
# More Open Preference Datasets:
We tried to stay as close as possible to the format proposed by the data-is-better-together community. This has the limitation that the quality of the style, coherence, and alignment are melted into one score. We have collected multiple datasets where we differentiate between these modalities:
- https://huggingface.co/collections/Rapidata/flux-sd3-mj-dalle-human-annotation-sets-675ae83c8ad7551e497e2c29
- https://huggingface.co/collections/Rapidata/rapidata-benchmark-data-675ae93f0193e1f84d66d083

If you have any questions, feel free to reach out to me at [e-mail]

<a href="https://www.rapidata.ai">
<img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="250" alt="Rapidata Logo">
</a>

我们希望为“数据共生（data-is-better-together）”社区提出的研究挑战贡献力量（社区说明如下）。我们通过自有API，在约3天内从全球各地的受访者处收集了170,000条偏好标注数据（详见docs.rapidata.ai）：

若您从本数据集获益并希望未来看到更多同类资源，欢迎为数据集点赞。

# 图像偏好结果数据集卡片 [原数据集](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1)

<style>
.row {
display: flex;
justify-content: space-between;
width: 100%;
}
#container {
display: flex;
flex-direction: column;
font-family: Arial, sans-serif;
width: 98%
}
.prompt {
margin-bottom: 10px;
font-size: 16px;
line-height: 1.4;
color: #333;
background-color: #f8f8f8;
padding: 10px;
border-radius: 5px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
}
.image-container {
display: flex;
gap: 10px;
}
.column {
flex: 1;
position: relative;
}
img {
max-width: 100%;
height: auto;
display: block;
}
.image-label {
position: absolute;
top: 10px;
right: 10px;
background-color: rgba(255, 255, 255, 0.7);
color: black;
padding: 5px 10px;
border-radius: 5px;
font-weight: bold;
}
</style>

<div class="row">
<div class="column">
<div id="container">
<div class="prompt"><strong>提示词：</strong> 玛雅羽蛇神生物改造体的动漫风格概念艺术，反乌托邦世界，色彩鲜艳，4K分辨率。</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1258.jpg">
<div class="image-label">图像1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1258.jpg">
<div class="image-label">图像2</div>
</div>
</div>
</div>
</div>
<div class="column">

<div id="container">
<div class="prompt"><strong>提示词：</strong> 8位像素风格作品，内容为蓝色骑士、绿色汽车与挪威冰川景观，奇幻风格，色彩丰富且细节饱满。</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1210.jpg">
<div class="image-label">图像1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1210.jpg">
<div class="image-label">图像2</div>
</div>
</div>
</div>
</div>
</div>

- **项目目标**：本项目旨在构建10,000组文本到图像（text-to-image）偏好配对样本。基于难度梯度各异的提示词，这些样本可用于评估图像生成模型在各类常见图像场景下的性能表现。

- **构建方式**：我们采用了[fal/imgsys-results](https://huggingface.co/datasets/fal/imgsys-results)数据集的提示词，这些提示词针对各类图像场景的复杂度与质量进行了优化迭代。随后我们邀请社区用户为每个提示词对应的两张生成图像标注偏好。

- **项目成果**：Rapidata从全球用户处收集了超过170,000条独立偏好标注。本次共涉及17,000组图像配对，每组配对均收集了约10条偏好标注。

- **标注流程**：标注者会收到问题：“根据上述描述，您更偏好哪一张图像？”，系统会同时展示提示词与两张待选图像，且不提供“两张都偏好”的选项。每组图像配对会随机打乱图像位置后分发给10位标注者，每条标注均附带标注者的可信度评分，该信息可在数据集的详细结果中查看。

- **数据格式**：本版本的结果已进行二值化处理。我们针对每组配对计算了基于用户评分的加权得分，并标注出被选中与被淘汰的图像。请注意，我们已剔除了没有明确偏好倾向的配对——我们将明确偏好定义为单张图像的加权用户得分占比不低于0.6。若您希望自定义阈值生成自己的二值化数据集，可访问[我们的原始数据集](https://huggingface.co/datasets/Rapidata/open-image-preferences-v1-more-results)获取所有原始标注信息。

# 更多开源偏好数据集：

我们尽可能遵循了“数据共生（data-is-better-together）”社区提出的数据集格式，但该格式存在一定局限：将图像风格质量、内容连贯性与语义对齐度的评估合并为单一得分。我们还收集了多组数据集，对上述不同维度的评估进行了区分：

- https://huggingface.co/collections/Rapidata/flux-sd3-mj-dalle-human-annotation-sets-675ae83c8ad7551e497e2c29
- https://huggingface.co/collections/Rapidata/rapidata-benchmark-data-675ae93f0193e1f84d66d083

若您有任何疑问，欢迎发送邮件至[e-mail]与我联系。

搜集汇总

背景与挑战

背景概述

该数据集由Rapidata收集，包含超过17万个基于文本提示的图像偏好标注，用于评估图像生成模型的性能。它基于10,000个文本-图像偏好对，每个图像对由约10名标注者进行偏好选择，并采用二值化处理以明确偏好结果。