image-preferences-results

# Dataset Card for image-preferences-results

<style>
.row {
display: flex;
justify-content: space-between;
width: 100%;
}
#container {
display: flex;
flex-direction: column;
font-family: Arial, sans-serif;
width: 98%
}
.prompt {
margin-bottom: 10px;
font-size: 16px;
line-height: 1.4;
color: #333;
background-color: #f8f8f8;
padding: 10px;
border-radius: 5px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
}
.image-container {
display: flex;
gap: 10px;
}
.column {
flex: 1;
position: relative;
}
img {
max-width: 100%;
height: auto;
display: block;
}
.image-label {
position: absolute;
top: 10px;
right: 10px;
background-color: rgba(255, 255, 255, 0.7);
color: black;
padding: 5px 10px;
border-radius: 5px;
font-weight: bold;
}
</style>

<div class="row">
<div class="column">
<div id="container">
<div class="prompt"><strong>Prompt:</strong> Anime-style concept art of a Mayan Quetzalcoatl biomutant, dystopian world, vibrant colors, 4K.</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1258.jpg">
<div class="image-label">Image 1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1258.jpg">
<div class="image-label">Image 2</div>
</div>
</div>
</div>
</div>
<div class="column">

<div id="container">
<div class="prompt"><strong>Prompt:</strong> 8-bit pixel art of a blue knight, green car, and glacier landscape in Norway, fantasy style, colorful and detailed.</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1210.jpg">
<div class="image-label">Image 1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1210.jpg">
<div class="image-label">Image 2</div>
</div>
</div>
</div>
</div>
</div>

- **Goal**: This project aims to create 10K text-to-image preference pairs. These pairs can be used to evaluate the performance of image generation models across a wide variety of common image categories, based on prompt with varying levels of difficulty.
- **How**: We use the prompts from [fal/imgsys-results](https://huggingface.co/datasets/fal/imgsys-results), these prompts are evolved based on complexity and quality for various image categories. We then asked the community to annotate the preference between two generated images for each prompt.
- **Result**: We achieved to annotate 10K preference pairs. You can take a look at the resulting dataset [here](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-results).

This dataset has been created with [Argilla](https://github.com/argilla-io/argilla). As shown in the sections below, this dataset can be loaded into your Argilla server as explained in [Load with Argilla](#load-with-argilla), or used directly with the `datasets` library in [Load with `datasets`](#load-with-datasets).

## Using this dataset with Argilla

To load with Argilla, you'll just need to install Argilla as `pip install argilla --upgrade` and then use the following code:

```python
import argilla as rg

ds = rg.Dataset.from_hub("data-is-better-together/image-preferences-results", settings="auto")
```

This will load the settings and records from the dataset repository and push them to you Argilla server for exploration and annotation.

## Using this dataset with `datasets`

To load the records of this dataset with `datasets`, you'll just need to install `datasets` as `pip install datasets --upgrade` and then use the following code:

```python
from datasets import load_dataset

ds = load_dataset("data-is-better-together/image-preferences-results")
```

This will only load the records of the dataset, but not the Argilla settings.

## Dataset Structure

This dataset repo contains:

* Dataset records in a format compatible with HuggingFace `datasets`. These records will be loaded automatically when using `rg.Dataset.from_hub` and can be loaded independently using the `datasets` library via `load_dataset`.
* The [annotation guidelines](#annotation-guidelines) that have been used for building and curating the dataset, if they've been defined in Argilla.
* A dataset configuration folder conforming to the Argilla dataset format in `.argilla`.

The dataset is created in Argilla with: **fields**, **questions**, **suggestions**, **metadata**, **vectors**, and **guidelines**.

### Fields

The **fields** are the features or text of a dataset's records. For example, the 'text' column of a text classification dataset of the 'prompt' column of an instruction following dataset.

| Field Name | Title | Type | Required | Markdown |
| ---------- | ----- | ---- | -------- | -------- |
| images | Images | custom | True | |

### Questions

The **questions** are the questions that will be asked to the annotators. They can be of different types, such as rating, text, label_selection, multi_label_selection, or ranking.

| Question Name | Title | Type | Required | Description | Values/Labels |
| ------------- | ----- | ---- | -------- | ----------- | ------------- |
| preference | Which image is better according to prompt adherence and aesthetics? | label_selection | True | Take a look at the guidelines (bottom left corner) to get more familiar with the project examples and our community. | ['image_1', 'image_2', 'both_good', 'both_bad', 'toxic_content'] |

<!-- check length of metadata properties -->

### Metadata

The **metadata** is a dictionary that can be used to provide additional information about the dataset record.
| Metadata Name | Title | Type | Values | Visible for Annotators |
| ------------- | ----- | ---- | ------ | ---------------------- |
| model_1 | model_1 | | - | True |
| model_2 | model_2 | | - | True |
| evolution | evolution | | - | True |

### Vectors
The **vectors** contain a vector representation of the record that can be used in search.

| Vector Name | Title | Dimensions |
|-------------|-------|------------|
| prompt | prompt | [1, 256] |

### Data Instances

An example of a dataset instance in Argilla looks as follows:

```json
{
"_server_id": "c2306976-5e44-4ad4-b2ce-8a510ec6086b",
"fields": {
"images": {
"image_1": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_dev/3368.jpg",
"image_2": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_sd/3368.jpg",
"prompt": "a bustling manga street, devoid of vehicles, detailed with vibrant colors and dynamic line work, characters in the background adding life and movement, under a soft golden hour light, with rich textures and a lively atmosphere, high resolution, sharp focus"
}
},
"id": "3368-quality",
"metadata": {
"category": "Manga",
"evolution": "quality",
"model_1": "dev",
"model_2": "sd",
"sub_category": "detailed"
},
"responses": {
"preference": [
{
"user_id": "50b9a890-173b-4999-bffa-fc0524ba6c63",
"value": "both_good"
},
{
"user_id": "caf19767-2989-4b3c-a653-9c30afc6361d",
"value": "image_1"
},
{
"user_id": "ae3e20b2-9aeb-4165-af54-69eac3f2448b",
"value": "image_1"
}
]
},
"status": "completed",
"suggestions": {},
"vectors": {}
}
```

While the same record in HuggingFace `datasets` looks as follows:

```json
{
"_server_id": "c2306976-5e44-4ad4-b2ce-8a510ec6086b",
"category": "Manga",
"evolution": "quality",
"id": "3368-quality",
"images": {
"image_1": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_dev/3368.jpg",
"image_2": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_sd/3368.jpg",
"prompt": "a bustling manga street, devoid of vehicles, detailed with vibrant colors and dynamic line work, characters in the background adding life and movement, under a soft golden hour light, with rich textures and a lively atmosphere, high resolution, sharp focus"
},
"model_1": "dev",
"model_2": "sd",
"preference.responses": [
"both_good",
"image_1",
"image_1"
],
"preference.responses.status": [
"submitted",
"submitted",
"submitted"
],
"preference.responses.users": [
"50b9a890-173b-4999-bffa-fc0524ba6c63",
"caf19767-2989-4b3c-a653-9c30afc6361d",
"ae3e20b2-9aeb-4165-af54-69eac3f2448b"
],
"prompt": null,
"status": "completed",
"sub_category": "detailed"
}
```

### Data Splits

The dataset contains a single split, which is `train`.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation guidelines

### Image Preference Task

We are aiming to collect preferences about images. We want to know which images are best in relation to another. So that we can train an AI model to generate images like the best ones.

### Your Contribution

Your task is to answer the question “Which image adheres best to the prompt?”. The prompt describes an image with objects, attributes, and styles. The images are generations with AI models based on the prompt.

### Keyboard shortcuts

Argilla offers [keyboard shortcuts](https://docs.argilla.io/dev/how_to_guides/annotate/#shortcuts), which will smoothen your annotation experience. TLDR: You can use numbers 1-5 to assign the corresponding labels, and press ENTER to submit.

### Definition of best image

The best image should contain all attributes of the prompt and be aesthetically pleasing in relation to the prompt.

**Attributes of the prompt** include objects, their attributes, and the style of the image. For example, *a realistic photograph of a red house with a dog in front of it.* The best image should contain each of these elements.

**Aesthetically pleasing** should relate to the prompt. If the prompt states a ‘realistic image’, then the best image would be the most realistic. If the prompt stated an ‘animated image’, then the best image would show the most appealing animation.

**Ties** are possible when both images do not meet either of the above criteria. For example, one image is unpleasant and the other does not adhere to the prompt. Or, both images meet all criteria perfectly.

### Example of scenarios

Example prompt: *A realistic photograph of a red house with a dog in front of it.*

<table>
<tr>
<th>Image 1</th>
<th>Image 2</th>
</tr>
<tr>
<td>image_1</td>
<td>Image_2 contains a yellow house, whilst Image_1 adheres to the prompt.</td>
</tr>
<tr>
<td>image_1</td>
<td><strong>Image_2 is an animation</strong>, whilst Image_1 adheres to the prompt.</td>
</tr>
<tr>
<td>image_1</td>
<td>Both adhere to the prompt, but <strong>image_2 is not aesthetically pleasing</strong>.</td>
</tr>
<tr>
<td>both</td>
<td>Both images follow the prompt completely, and there is no aesthetic difference.</td>
</tr>
<tr>
<td>neither</td>
<td>Neither image follows the prompts.</td>
</tr>
<tr>
<td>neither</td>
<td>Image_2 contains all aspects mentioned in the prompt, but is not aesthetically pleasing. Image_1 does not adhere to the prompt.</td>
</tr>
<tr>
<td>Toxic ⚠️</td>
<td>Any content that is <strong>Not suitable for work.</strong> For example, sexualized or offensive images.</td>
</tr>
</table>

### Socials, leaderboards and discussions

This is a community event so discussion and sharing are encouraged. We are available in the [#data-is-better-together channel on the Hugging Face discord](https://discord.com/channels/879548962464493619/1205128865735770142), on [@argilla_io on X](https://x.com/argilla_io) and as [@Argilla LinkedIn](https://www.linkedin.com/company/11501021/admin/dashboard/) too. Lastly, you can follow [our Hugging Face organisation](https://huggingface.co/data-is-better-together) and we've got a [progress leaderboard](https://huggingface.co/spaces/data-is-better-together/image-preferences-leaderboard) that will be used for prices.

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]

# 图像偏好结果数据集（image-preferences-results）

<style>
.row {
display: flex;
justify-content: space-between;
width: 100%;
}
#container {
display: flex;
flex-direction: column;
font-family: Arial, sans-serif;
width: 98%
}
.prompt {
margin-bottom: 10px;
font-size: 16px;
line-height: 1.4;
color: #333;
background-color: #f8f8f8;
padding: 10px;
border-radius: 5px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
}
.image-container {
display: flex;
gap: 10px;
}
.column {
flex: 1;
position: relative;
}
img {
max-width: 100%;
height: auto;
display: block;
}
.image-label {
position: absolute;
top: 10px;
right: 10px;
background-color: rgba(255, 255, 255, 0.7);
color: black;
padding: 5px 10px;
border-radius: 5px;
font-weight: bold;
}
</style>

<div class="row">
<div class="column">
<div id="container">
<div class="prompt"><strong>提示词：</strong> 玛雅羽蛇神生物改造体的动漫风格概念艺术，反乌托邦世界，色彩鲜艳，4K分辨率。</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1258.jpg">
<div class="image-label">图像1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1258.jpg">
<div class="image-label">图像2</div>
</div>
</div>
</div>
</div>
<div class="column">

<div id="container">
<div class="prompt"><strong>提示词：</strong> 8位像素艺术风格作品，内容为蓝色骑士、绿色汽车与挪威冰川景观，奇幻风格，色彩丰富且细节饱满。</div>
<div class="image-container">
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_dev/1210.jpg">
<div class="image-label">图像1</div>
</div>
<div class="column">
<img src="https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1/resolve/main/image_simplified_sd/1210.jpg">
<div class="image-label">图像2</div>
</div>
</div>
</div>
</div>
</div>

## 项目目标
本项目旨在构建10000组文本到图像的偏好配对样本。基于不同难度梯度的提示词，这些样本可用于评估图像生成模型在各类常见图像类别上的性能表现。

## 构建方式
我们选用了[fal/imgsys-results](https://huggingface.co/datasets/fal/imgsys-results)数据集中的提示词，这些提示词针对各类图像类别，基于复杂度与质量进行了优化升级。随后我们邀请社区用户为每个提示词对应的两张生成图像标注偏好等级。

## 项目成果
我们已完成10000组偏好配对样本的标注工作。你可以通过[此链接](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-results)查看最终生成的数据集。

本数据集基于[Argilla](https://github.com/argilla-io/argilla)工具构建。如下文所述，你既可以按照[通过Argilla加载](#load-with-argilla)的步骤将数据集导入你的Argilla服务器，也可以直接通过`datasets`库加载，详见[通过`datasets`加载](#load-with-datasets)。

## 使用Argilla加载数据集
你只需通过`pip install argilla --upgrade`安装Argilla，随后运行以下代码即可：
python
import argilla as rg

ds = rg.Dataset.from_hub("data-is-better-together/image-preferences-results", settings="auto")

该代码将从数据集仓库加载配置与样本数据，并将其推送至你的Argilla服务器，以供探索与标注使用。

## 使用`datasets`库加载数据集
你只需通过`pip install datasets --upgrade`安装`datasets`库，随后运行以下代码即可：
python
from datasets import load_dataset

ds = load_dataset("data-is-better-together/image-preferences-results")

该代码仅会加载数据集的样本数据，不会加载Argilla相关配置。

## 数据集结构
本数据集仓库包含以下内容：
* 适配HuggingFace `datasets`格式的数据集样本。使用`rg.Dataset.from_hub`时将自动加载此类样本，也可通过`datasets`库的`load_dataset`方法独立加载。
* 数据集标注指南（若已在Argilla中定义），用于数据集的构建与审核。
* 符合Argilla数据集格式的`.argilla`配置文件夹。

本数据集在Argilla中通过以下模块构建：**字段（fields）**、**问题（questions）**、**建议（suggestions）**、**元数据（metadata）**、**向量（vectors）**与**指南（guidelines）**。

### 字段
**字段**指数据集样本的特征或文本内容。例如，文本分类数据集的`text`列，或指令跟随数据集的`prompt`列。

| 字段名 | 标题 | 类型 | 必填 | Markdown支持 |
| ---------- | ----- | ---- | -------- | -------- |
| images | 图像 | 自定义 | 是 | 否 |

### 问题
**问题**指向标注者提出的查询任务，支持评分、文本、标签选择、多标签选择、排序等多种类型。

| 问题名 | 标题 | 类型 | 必填 | 描述 | 可选值/标签 |
| ------------- | ----- | ---- | -------- | ----------- | ------------- |
| preference | 根据提示词贴合度与美学效果，哪张图像更优？ | 标签选择 | 是 | 请查看标注指南（左下角）以熟悉本项目示例与社区规则。 | ['image_1', 'image_2', 'both_good', 'both_bad', 'toxic_content'] |

<!-- check length of metadata properties -->

### 元数据
**元数据**指用于提供数据集样本额外信息的字典结构。
| 元数据项名 | 标题 | 类型 | 可选值 | 对标注者可见 |
| ------------- | ----- | ---- | ------ | ---------------------- |
| model_1 | 模型1 | | - | 是 |
| model_2 | 模型2 | | - | 是 |
| evolution | 迭代优化方向 | | - | 是 |

### 向量
**向量**指用于检索的样本向量表示。

| 向量名 | 标题 | 维度 |
|-------------|-------|------------|
| prompt | 提示词 | [1, 256] |

### 数据样本示例
Argilla格式的数据集样本示例如下：
json
{
"_server_id": "c2306976-5e44-4ad4-b2ce-8a510ec6086b",
"fields": {
"images": {
"image_1": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_dev/3368.jpg",
"image_2": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_sd/3368.jpg",
"prompt": "a bustling manga street, devoid of vehicles, detailed with vibrant colors and dynamic line work, characters in the background adding life and movement, under a soft golden hour light, with rich textures and a lively atmosphere, high resolution, sharp focus"
}
},
"id": "3368-quality",
"metadata": {
"category": "漫画",
"evolution": "质量优化",
"model_1": "dev",
"model_2": "sd",
"sub_category": "细节丰富"
},
"responses": {
"preference": [
{
"user_id": "50b9a890-173b-4999-bffa-fc0524ba6c63",
"value": "both_good"
},
{
"user_id": "caf19767-2989-4b3c-a653-9c30afc6361d",
"value": "image_1"
},
{
"user_id": "ae3e20b2-9aeb-4165-af54-69eac3f2448b",
"value": "image_1"
}
]
},
"status": "已完成",
"suggestions": {},
"vectors": {}
}

而HuggingFace `datasets`格式的同一样本示例如下：
json
{
"_server_id": "c2306976-5e44-4ad4-b2ce-8a510ec6086b",
"category": "漫画",
"evolution": "质量优化",
"id": "3368-quality",
"images": {
"image_1": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_dev/3368.jpg",
"image_2": "https://huggingface.co/datasets/data-is-better-together/image-preferences-filtered/resolve/main/image_quality_sd/3368.jpg",
"prompt": "a bustling manga street, devoid of vehicles, detailed with vibrant colors and dynamic line work, characters in the background adding life and movement, under a soft golden hour light, with rich textures and a lively atmosphere, high resolution, sharp focus"
},
"model_1": "dev",
"model_2": "sd",
"preference.responses": [
"both_good",
"image_1",
"image_1"
],
"preference.responses.status": [
"submitted",
"submitted",
"submitted"
],
"preference.responses.users": [
"50b9a890-173b-4999-bffa-fc0524ba6c63",
"caf19767-2989-4b3c-a653-9c30afc6361d",
"ae3e20b2-9aeb-4165-af54-69eac3f2448b"
],
"prompt": null,
"status": "已完成",
"sub_category": "细节丰富"
}

### 数据拆分
本数据集仅包含一个拆分集：`train`（训练集）。

## 数据集创建
### 策划初衷
[需补充更多信息]

### 源数据
#### 初始数据收集与标准化
[需补充更多信息]

#### 源语言生成者是谁？
[需补充更多信息]

### 标注信息
#### 标注指南
##### 图像偏好标注任务
我们旨在收集图像偏好数据，以明确不同图像间的优劣关系，进而训练AI模型生成更优质的图像。

##### 标注者任务
你的任务是回答“哪张图像最贴合给定提示词？”。提示词描述了图像应包含的对象、属性与风格，而两张图像均为基于该提示词通过AI模型生成的作品。

##### 键盘快捷键
Argilla支持[键盘快捷键](https://docs.argilla.io/dev/how_to_guides/annotate/#shortcuts)，可优化你的标注体验。简而言之：你可使用数字1-5对应选择相应标签，按下回车提交结果。

##### 最优图像的定义
最优图像需同时满足两点要求：一是完整包含提示词中的所有属性，二是相对于提示词要求具备最佳美学效果。
- **提示词属性**：包括对象、对象属性与图像风格。例如，“一张红色房屋前带有狗狗的写实照片”，最优图像需包含所有上述元素。
- **美学效果**：需与提示词要求匹配。若提示词要求“写实照片”，则最优图像应为最写实的作品；若提示词要求“动画风格”，则最优图像应为观感最佳的动画作品。
- **平局情况**：当两张图像均不符合上述标准时，可判定为平局。例如，一张图像观感糟糕，另一张未贴合提示词；或两张图像均完美符合所有标准。

##### 场景示例
示例提示词：*一张红色房屋前带有狗狗的写实照片*。
<table>
<tr>
<th>图像1</th>
<th>图像2</th>
</tr>
<tr>
<td>图像1符合提示词</td>
<td>图像2的房屋为黄色，未贴合提示词。</td>
</tr>
<tr>
<td>图像1符合提示词</td>
<td>图像2为动画风格，未贴合提示词的写实要求。</td>
</tr>
<tr>
<td>两张图像均贴合提示词，但图像2美学效果较差。</td>
<td>图像2</td>
</tr>
<tr>
<td>两张图像均完美贴合提示词且无美学差异。</td>
<td>平局</td>
</tr>
<tr>
<td>两张图像均未贴合提示词。</td>
<td>平局</td>
</tr>
<tr>
<td>图像2包含提示词所有元素但美学效果较差，图像1未贴合提示词。</td>
<td>平局</td>
</tr>
<tr>
<td>不适宜内容 ⚠️</td>
<td>任何不适宜公开的内容，例如色情或冒犯性图像。</td>
</tr>
</table>

##### 社群交流、排行榜与讨论
本项目属于社区共建活动，鼓励讨论与经验分享。你可通过以下渠道联系我们：
- Hugging Face Discord频道[#data-is-better-together](https://discord.com/channels/879548962464493619/1205128865735770142)
- X平台[@argilla_io](https://x.com/argilla_io)
- LinkedIn官方账号[Argilla](https://www.linkedin.com/company/11501021/admin/dashboard/)
此外，你可以关注我们的[Hugging Face组织主页](https://huggingface.co/data-is-better-together)，并通过[进度排行榜](https://huggingface.co/spaces/data-is-better-together/image-preferences-leaderboard)查看项目进展与获奖情况。

#### 标注流程
[需补充更多信息]

#### 标注者是谁？
[需补充更多信息]

### 个人与敏感信息
[需补充更多信息]

## 数据使用注意事项
### 数据集的社会影响
[需补充更多信息]

### 偏差讨论
[需补充更多信息]

### 其他已知局限
[需补充更多信息]

## 补充信息
### 数据集策划者
[需补充更多信息]

### 授权信息
[需补充更多信息]

### 引用信息
[需补充更多信息]

### 贡献者
[需补充更多信息]