ImagenWorld-condition-set

## 📦 Dataset Access

The dataset contains **zipped folders** for each task. You can download and extract the dataset in **two ways**:

---

### 🐍 **Option 1 — Python**

```python
from huggingface_hub import snapshot_download
import os
import zipfile
from pathlib import Path

# Download dataset
local_path = snapshot_download(
repo_id="TIGER-Lab/ImagenWorld-condition-set",
repo_type="dataset",
local_dir="ImagenWorld-condition-set", # 👈 where files will be saved
local_dir_use_symlinks=False
)

print("Files saved to:", local_path)

# Unzip all task folders
for zip_file in Path(local_path).glob("*.zip"):
target_dir = Path(local_path) / zip_file.stem
target_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(zip_file, 'r') as zf:
zf.extractall(target_dir)
print(f"✅ Extracted {zip_file.name} to {target_dir}")
```

---

### 💻 **Option 2 — Command Line (one-liners)**

#### **Download**
```bash
hf dataset download TIGER-Lab/ImagenWorld-condition-set --repo-type dataset --local-dir ImagenWorld-condition-set
```

#### **Unzip all**
```bash
cd ImagenWorld-condition-set && for f in *.zip; do d="${f%.zip}"; mkdir -p "$d"; unzip -q "$f" -d "$d"; done
```

After extraction, your structure will look like this:

```
imagenworld_local/
│
├── TIG/
├── TIE/
├── SRIG/
├── SRIE/
├── MRIG/
└── MRIE/
```

---

## 📁 Dataset Structure

Each task folder (e.g., `TIG`, `TIE`, `SRIG`, `SRIE`, `MRIG`, `MRIE`) contains multiple entries.
Each entry corresponds to a single condition set — identified by a unique ID such as `TIG_A_000001`.

```
ImagenWorld/
│
├── TIG/ # Text-to-Image Generation
│ ├── TIG_A_000001/
│ │ ├── metadata.json # Task metadata and prompt
│ │ ├── 1.png # reference images
│ │ ├── 2.png
│ │ └── ...
│ └── ...
│
├── TIE/ # Text + Image Editing
├── SRIG/ # Single-Reference Image Generation
├── SRIE/ # Single-Reference Image Editing
├── MRIG/ # Multi-Reference Image Generation
└── MRIE/ # Multi-Reference Image Editing
```

Each `metadata.json` includes:
```json
{
"task": "<Task type — one of: TIG, TIE, SRIG, SRIE, MRIG, MRIE>",
"topic": "<Domain ID — representing one of the six visual domains>",
"subtopic": "<Specific visual style or subdomain, e.g., 'Oil painting', 'UI mockup', 'Medical diagram'>",

"prompt": "<Original text instruction provided to the model>",
"cond_images": [
"<List of condition or reference image filenames, e.g., '1.jpg', '2.jpg', ...>"
],

"remarks": "<Optional free-text notes from annotators (may be empty)>",
"prompt_refined": "<Refined or standardized version of the prompt for reproducibility>",
"annotator": "<Annotator name or ID>",

"objects": "<List of objects expected to appear in the model-generated image> (generated by vlm)",

"human_annotation": "<Boolean — specifies whether this entry has been annotated for object- and segment-level issues for closed-source models (e.g., gpt-image-1, gemini-2-flash)>",
"human_annotation_opensource": "<Boolean — specifies whether this entry has been annotated for object- and segment-level issues for open-source models (e.g., SDXL, OmniGeni2)>"
}

```

---

## 🧩 Tasks Overview

| Task | Name | Description |
|------|------|--------------|
| **TIG** | Text-to-Image Generation | Generate an image purely from a textual description. |
| **TIE** | Text and Image Editing | Edit a given image based on a textual instruction. |
| **SRIG** | Single-Reference Image Generation | Generate an image using a single reference image and a text prompt. |
| **SRIE** | Single-Reference Image Editing | Edit an image using both a text prompt and a single reference. |
| **MRIG** | Multi-Reference Image Generation | Generate new images using multiple references and text. |
| **MRIE** | Multi-Reference Image Editing | Edit an image using multiple references and text. |

---

## 🎨 Domains

Each task covers six **visual domains**, ensuring cross-domain robustness:

1. **Artworks (A)**
2. **Photorealistic Images (p)**
3. **Information Graphics (I)**
4. **Textual Graphics (T)**
5. **Computer Graphics (C)**
6. **Screenshots (S)**
---

## 📦 Dataset Stats

| Property | Value |
|-----------|--------|
| Total Tasks | 6 |
| Total Topics | 6 |
| Total Condition Sets | ~3.6K |
| Annotation Type | Human-written text |

---

## 🔗 Related Datasets

| Component | Description | Repository |
|------------|--------------|-------------|
| **Model Outputs** | Generated images from open- and closed-source models evaluated on ImagenWorld. | [`TIGER-Lab/ImagenWorld-model-outputs`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-model-outputs) |
| **Annotated Set** | Includes both `train` and `test` splits — only `train` contains human annotations; the test split is simply the remaining portion without manual evaluation. | [`TIGER-Lab/ImagenWorld-annotated-set`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-annotated-set) |

---
## 📜 Citation

If you use **ImagenWorld**, please cite:

```bibtex
@misc{imagenworld2025,
title = {ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks},
author = {Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Ting Wai Lam and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen},
year = {2025},
doi = {10.5281/zenodo.17344183},
url = {https://zenodo.org/records/17344183},
projectpage = {https://tiger-ai-lab.github.io/ImagenWorld/},
blogpost = {https://blog.comfy.org/p/introducing-imagenworld},
note = {Community-driven dataset and benchmark release, Temporarily archived on Zenodo while arXiv submission is under moderation review.},
}

```

---

📦 数据集获取方式

本数据集为每个任务封装为**压缩包文件夹**。您可通过以下两种方式下载并解压本数据集：

---

### 🐍 **选项一 — Python 代码方式**

python
from huggingface_hub import snapshot_download
import os
import zipfile
from pathlib import Path

# 下载数据集
local_path = snapshot_download(
repo_id="TIGER-Lab/ImagenWorld-condition-set",
repo_type="dataset",
local_dir="ImagenWorld-condition-set", # 👈 文件将保存至该目录
local_dir_use_symlinks=False
)

print("文件已保存至：", local_path)

# 解压所有任务压缩包
for zip_file in Path(local_path).glob("*.zip"):
target_dir = Path(local_path) / zip_file.stem
target_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(zip_file, 'r') as zf:
zf.extractall(target_dir)
print(f"✅ 已将 {zip_file.name} 解压至 {target_dir}")

---

### 💻 **选项二 — 命令行（单行命令）**

#### **下载**
bash
hf dataset download TIGER-Lab/ImagenWorld-condition-set --repo-type dataset --local-dir ImagenWorld-condition-set

#### **批量解压**
bash
cd ImagenWorld-condition-set && for f in *.zip; do d="${f%.zip}"; mkdir -p "$d"; unzip -q "$f" -d "$d"; done

解压完成后，数据集目录结构如下：

imagenworld_local/
│
├── TIG/
├── TIE/
├── SRIG/
├── SRIE/
├── MRIG/
└── MRIE/

---

## 📁 数据集组织结构

每个任务文件夹（如`TIG`、`TIE`、`SRIG`、`SRIE`、`MRIG`、`MRIE`）均包含多个条目。每个条目对应一个独立的条件集，以唯一ID标识，例如`TIG_A_000001`。

ImagenWorld/
│
├── TIG/ # 文本到图像生成（Text-to-Image Generation）
│ ├── TIG_A_000001/
│ │ ├── metadata.json # 任务元数据与提示词
│ │ ├── 1.png # 参考图像
│ │ ├── 2.png
│ │ └── ...
│ └── ...
│
├── TIE/ # 文本与图像编辑（Text + Image Editing）
├── SRIG/ # 单参考图像生成（Single-Reference Image Generation）
├── SRIE/ # 单参考图像编辑（Single-Reference Image Editing）
├── MRIG/ # 多参考图像生成（Multi-Reference Image Generation）
└── MRIE/ # 多参考图像编辑（Multi-Reference Image Editing）

每个`metadata.json`文件包含以下字段：
json
{
"任务": "<任务类型——可选值为：TIG、TIE、SRIG、SRIE、MRIG、MRIE>",
"主题域": "<领域ID，对应六大视觉域之一>",
"子主题": "<具体视觉风格或子领域，例如：'油画'、'UI原型图'、'医学示意图'>",

"提示词": "<模型接收的原始文本指令>",
"条件图像列表": [
"<条件图像或参考图像的文件名列表，例如：'1.jpg'、'2.jpg'……>"
],

"备注": "<标注人员撰写的可选自由文本注释（可为空）>",
"优化后提示词": "<为保证可复现性而优化或标准化后的提示词>",
"标注人员": "<标注人员姓名或ID>",

"预期物体列表": "<模型生成图像中应包含的物体列表（由视觉语言模型生成）>",

"人工标注状态": "<布尔值，表明该条目是否已针对闭源模型（如gpt-image-1、gemini-2-flash）的物体与片段级问题完成人工标注>",
"开源模型人工标注状态": "<布尔值，表明该条目是否已针对开源模型（如SDXL、OmniGeni2）的物体与片段级问题完成人工标注>"
}

---

## 🧩 任务概览

| 任务代号 | 任务名称 | 任务描述 |
|------|------|--------------|
| **TIG** | 文本到图像生成（Text-to-Image Generation） | 仅根据文本描述生成图像。 |
| **TIE** | 文本与图像编辑（Text and Image Editing） | 根据文本指令编辑给定图像。 |
| **SRIG** | 单参考图像生成（Single-Reference Image Generation） | 使用单张参考图像与文本提示生成图像。 |
| **SRIE** | 单参考图像编辑（Single-Reference Image Editing） | 结合文本提示与单张参考图像编辑图像。 |
| **MRIG** | 多参考图像生成（Multi-Reference Image Generation） | 使用多张参考图像与文本提示生成新图像。 |
| **MRIE** | 多参考图像编辑（Multi-Reference Image Editing） | 结合多张参考图像与文本提示编辑图像。 |

---

## 🎨 视觉域

本数据集覆盖六大**视觉域**，以保证模型的跨域鲁棒性：

1. **艺术作品（Artworks，缩写A）**
2. **写实摄影图像（Photorealistic Images，缩写p）**
3. **信息图表（Information Graphics，缩写I）**
4. **文本图形（Textual Graphics，缩写T）**
5. **计算机图形（Computer Graphics，缩写C）**
6. **屏幕截图（Screenshots，缩写S）**
---

## 📦 数据集统计信息

| 属性 | 数值 |
|-----------|--------|
| 总任务类型数 | 6 |
| 总主题域数 | 6 |
| 总条件集数量 | 约3.6K |
| 标注类型 | 人工文本标注 |

---

## 🔗 关联数据集

| 组件 | 描述 | 仓库地址 |
|------------|--------------|-------------|
| **模型输出集** | 开源与闭源模型在ImagenWorld上生成的评估图像 | [`TIGER-Lab/ImagenWorld-model-outputs`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-model-outputs) |
| **标注数据集** | 包含训练集与测试集划分——仅训练集带有人工标注；测试集为未经过人工评估的剩余样本 | [`TIGER-Lab/ImagenWorld-annotated-set`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-annotated-set) |

---
## 📜 引用格式

若您使用本ImagenWorld数据集，请引用如下文献：

bibtex
@misc{imagenworld2025,
title = {ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks},
author = {Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Ting Wai Lam and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen},
year = {2025},
doi = {10.5281/zenodo.17344183},
url = {https://zenodo.org/records/17344183},
projectpage = {https://tiger-ai-lab.github.io/ImagenWorld/},
blogpost = {https://blog.comfy.org/p/introducing-imagenworld},
note = {Community-driven dataset and benchmark release, Temporarily archived on Zenodo while arXiv submission is under moderation review.},
}