HPDv2

# Human Preference Dataset v2 (HPD v2)

**Human Preference Dataset v2 (HPD v2)** is a large-scale, cleanly-annotated dataset of human preferences for images generated from text prompts.

For more detailed information, please refer to the paper: [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://arxiv.org/abs/2306.09341).

We also trained [Human Preference Score v2 (HPSv2)](https://github.com/tgxs002/HPSv2), a preference prediction model, on HPD v2.

## Updates

* [07/29/2023] We released the benchmark and HPD v2 test data. HPD v2 train data will be **released sonn**.

## Data Source


The prompts in our dataset are sourced from DiffusionDB and MSCOCO Captions. Prompts from DiffusionDB are first cleaned by ChatGPT to remove biased function words. Human annotators are tasked to rank images generated by different text-to-image generative models from the same prompt. Totally there are about 798k pairwise comparisons of images for over 430k images and 107k prompts, 645k pairs for training split and 153k pairs for test split.

Image sources of HPD v2:
| Source | # of images
| :-----: | :-----: |
| CogView2 | 73697 |
| DALL·E 2 | 101869 |
| GLIDE (mini) | 400 |
| Stable Diffusion v1.4 | 101869 |
| Stable Diffusion v2.0 | 101869 |
| LAFITE | 400 |
| VQ-GAN+CLIP | 400 |
| VQ-Diffusion | 400 |
| FuseDream | 400 |
| COCO Captions | 28272 |

# Evaluation prompts

We also provide a set of evaluation prompts (benchmark prompts) that involves testing a model on a total of 3200 prompts, with 800 prompts for each of the following styles: “Animation”, “Concept-art”, “Painting”, and “Photo”.

In this reposity, We include benchmark images generated by mainstream text-to-image generative model based on benchmark prompts. So far, the following models have been included (being continuously updated):

- ChilloutMix
- CogView2
- DALL·E mini
- DALL·E 2
- Deliberate
- DeepFloyd-XL
- Dreamlike Photoreal 2.0
- Epic Diffusion
- FuseDream
- GLIDE
- LAFITE
- Latent Diffusion
- MajicMix Realistic
- Openjourney
- Realistic Vision
- Stable Diffusion v1.4
- Stable Diffusion v2.0
- SDXL Base 0.9
- SDXL Refiner 0.9
- Versatile Diffusion
- VQ-Diffusion
- VQGAN + CLIP

## Structure

Once unzipped, you should get a folder with the following structure:
```
HPD
---- train/
-------- {image_id}.jpg
---- test/
-------- {image_id}.jpg
---- train.json
---- test.json
---- benchmark/
-------- benchmark_imgs/
------------ {model_id}/
---------------- {image_id}.jpg
-------- drawbench/
------------ {model_id}/
---------------- {image_id}.jpg
-------- anime.json
-------- concept-art.json
-------- paintings.json
-------- photo.json
-------- drawbench.json
```

The annotation file, `train.json`, is organized as:
```
[
{
'human_preference': list[int], # 1 for preference
'prompt': str,
'file_path': list[str],
'user_hash': str,
},
...
]
```

The annotation file, `test.json`, is organized as:
```
[
{
'prompt': str,
'image_path': list[str],
'rank': list[int], # ranking for image at the same index in image_path
},
...
]
```

The benchmark prompts file, ie. `anime.json` is pure prompts. The corresponding image can be found in the folder of the corresponding model by indexing the prompt.

# 人类偏好数据集v2（Human Preference Dataset v2, HPD v2）

**人类偏好数据集v2（Human Preference Dataset v2, HPD v2）** 是一款大规模、标注严谨的文本提示生成图像人类偏好数据集。

如需获取更详细信息，请参阅论文：《Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis》，链接：https://arxiv.org/abs/2306.09341。

我们还基于HPD v2训练了**人类偏好评分v2（Human Preference Score v2, HPSv2）**——一款偏好预测模型，其开源仓库地址为：https://github.com/tgxs002/HPSv2。

## 更新日志
* [2023年7月29日] 本基准测试集与HPD v2测试数据已正式发布。HPD v2训练数据将**即将发布**。

## 数据来源

本数据集的提示词源自DiffusionDB与MSCOCO Captions。其中来自DiffusionDB的提示词首先经由ChatGPT进行清洗，移除存在偏差的功能词。人类标注员的任务是：针对同一文本提示生成的不同文本到图像生成模型输出的图像进行排序。本数据集总计包含约79.8万组图像成对比较样本，覆盖超43万张图像与10.7万个提示词；其中训练集包含64.5万组样本，测试集包含15.3万组样本。

### HPD v2图像来源
| 图像来源 | 图像数量 |
| :-----: | :-----: |
| CogView2 | 73697 |
| DALL·E 2 | 101869 |
| GLIDE (mini) | 400 |
| Stable Diffusion v1.4 | 101869 |
| Stable Diffusion v2.0 | 101869 |
| LAFITE | 400 |
| VQ-GAN+CLIP | 400 |
| VQ-Diffusion | 400 |
| FuseDream | 400 |
| COCO Captions | 28272 |

## 评估提示词
我们还提供了一套评估提示词（基准测试提示词），总计包含3200个测试提示词，分为以下四大风格，每个风格各800个提示词：「动画（Animation）」、「概念艺术（Concept-art）」、「绘画（Painting）」与「摄影（Photo）」。

本仓库中包含了基于基准测试提示词、由主流文本到图像生成模型生成的基准图像。截至目前，已包含以下模型（后续将持续更新）：
- ChilloutMix
- CogView2
- DALL·E mini
- DALL·E 2
- Deliberate
- DeepFloyd-XL
- Dreamlike Photoreal 2.0
- Epic Diffusion
- FuseDream
- GLIDE
- LAFITE
- Latent Diffusion
- MajicMix Realistic
- Openjourney
- Realistic Vision
- Stable Diffusion v1.4
- Stable Diffusion v2.0
- SDXL Base 0.9
- SDXL Refiner 0.9
- Versatile Diffusion
- VQ-Diffusion
- VQGAN + CLIP

## 数据集结构
解压后将得到如下目录结构：

HPD
---- train/
-------- {image_id}.jpg
---- test/
-------- {image_id}.jpg
---- train.json
---- test.json
---- benchmark/
-------- benchmark_imgs/
------------ {model_id}/
---------------- {image_id}.jpg
-------- drawbench/
------------ {model_id}/
---------------- {image_id}.jpg
-------- anime.json
-------- concept-art.json
-------- paintings.json
-------- photo.json
-------- drawbench.json

注释文件`train.json`的组织格式如下：

[
{
'human_preference': list[int], # 1代表该图像为偏好结果
'prompt': str,
'file_path': list[str],
'user_hash': str,
},
...
]

注释文件`test.json`的组织格式如下：

[
{
'prompt': str,
'image_path': list[str],
'rank': list[int], # 与image_path列表中同索引位置的图像相对应的排序结果
},
...
]

基准测试提示词文件（如`anime.json`）仅包含纯提示词文本，对应的图像可通过提示词索引至对应模型的文件夹中找到。