ImagenWorld-model-outputs

## 📦 Dataset Access

The dataset is organized as **zipped folders**, one per task.
Each task folder contains multiple condition sets, and each condition set folder contains two subfolders:
- `input/` — the original condition set (metadata and reference images)
- `model_output/` — the generated outputs from all included models

---

### 🐍 **Download with Python**

```python
from huggingface_hub import snapshot_download
import zipfile
from pathlib import Path

# Download model outputs
local_path = snapshot_download(
repo_id="TIGER-Lab/ImagenWorld-model-outputs",
repo_type="dataset",
local_dir="ImagenWorld-model-outputs",
local_dir_use_symlinks=False,
)

# Unzip all tasks
for zip_file in Path(local_path).glob("*.zip"):
target_dir = Path(local_path) / zip_file.stem
target_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(zip_file, "r") as zf:
zf.extractall(target_dir)
print(f"✅ Extracted {zip_file.name} → {target_dir}")
```

---

### 💻 **Download via Command Line**

```bash
hf dataset download TIGER-Lab/ImagenWorld-model-outputs --repo-type dataset --local-dir ImagenWorld-model-outputs
cd ImagenWorld-model-outputs && for f in *.zip; do d="${f%.zip}"; mkdir -p "$d"; unzip -q "$f" -d "$d"; done
```

---

## 📁 Dataset Structure

After extraction, your directory will look like this:

```
ImagenWorld-model-outputs/
│
├── TIG/
│ ├── TIG_A_000001/
│ │ ├── input/
│ │ │ ├── metadata.json # task metadata, prompt, and references
│ │ │ ├── 1.png # reference or condition image(s)
│ │ │ └── ...
│ │ └── model_output/
│ │ ├── sdxl.png # model output for SDXL
│ │ ├── gpt-image-1.png # model output for GPT-Image-1
│ │ ├── gemini.png # model output for Gemini 2.0 Flash
│ │ └── ...
│ └── ...
│
├── TIE/
├── SRIG/
├── SRIE/
├── MRIG/
└── MRIE/
```

---

## 🧠 Included Models

Below are the models included for each ImagenWorld task:

- **TIG (Text-to-Image Generation)**
SDXL, Infinity, Janus Pro, GPT-Image-1, UNO, BAGEL, Gemini 2.0 Flash, OmniGen 2, Flux.1-Krea-dev, Qwen-Image , Nano Banana

- **TIE (Text + Image Editing)**
InstructPix2Pix, GPT-Image-1, BAGEL, Step1X-Edit, IC-Edit, Gemini 2.0 Flash, OmniGen 2, Flux.1-Kontext-dev , Nano Banana

- **SRIG (Single-Reference Image Generation)**
GPT-Image-1, Gemini 2.0 Flash, OmniGen 2, BAGEL, UNO, Nano Banana

- **SRIE (Single-Reference Image Editing)**
GPT-Image-1, Gemini 2.0 Flash, OmniGen 2, BAGEL, Nano Banana

- **MRIG (Multi-Reference Image Generation)**
GPT-Image-1, Gemini 2.0 Flash, OmniGen 2, BAGEL, UNO, Nano Banana

- **MRIE (Multi-Reference Image Editing)**
GPT-Image-1, Gemini 2.0 Flash, OmniGen 2, BAGEL, Nano Banana

Each folder within `model_output/` contains images named after these models, e.g.:

```
model_output/
├── sdxl.png
├── gpt-image-1.png
├── gemini.png
└── ...
```

---

## 🧩 Tasks Overview

| Task | Name | Description |
|------|------|--------------|
| **TIG** | Text-to-Image Generation | Generate an image purely from a textual description. |
| **TIE** | Text and Image Editing | Edit a given image based on a textual instruction. |
| **SRIG** | Single-Reference Image Generation | Generate an image using a single reference image and a text prompt. |
| **SRIE** | Single-Reference Image Editing | Edit an image using both a text prompt and a single reference. |
| **MRIG** | Multi-Reference Image Generation | Generate images using multiple references and text. |
| **MRIE** | Multi-Reference Image Editing | Edit an image using multiple references and text. |

---

## 🎨 Domains

Each task spans six **visual domains**, ensuring cross-domain robustness:

1. **Artworks (A)**
2. **Photorealistic Images (P)**
3. **Information Graphics (I)**
4. **Textual Graphics (T)**
5. **Computer Graphics (C)**
6. **Screenshots (S)**

---

## 🔗 Related Datasets

| Component | Description | Repository |
|------------|--------------|-------------|
| **Condition Set** | Input prompts, metadata, and reference images. | [`TIGER-Lab/ImagenWorld`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld) |
| **Annotated Set** | Includes both `train` and `test` splits — only `train` contains human annotations; the test split is simply the remaining portion without manual evaluation. | [`TIGER-Lab/ImagenWorld-annotated-set`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-annotated-set) |

---

## 📜 Citation

If you use **ImagenWorld**, please cite:

```bibtex
@misc{imagenworld2025,
title = {ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks},
author = {Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Ting Wai Lam and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen},
year = {2025},
doi = {10.5281/zenodo.17344183},
url = {https://zenodo.org/records/17344183},
note = {Community-driven dataset and benchmark release, temporarily archived on Zenodo while arXiv submission is under review.},
}
```

## 📦 数据集获取
本数据集以**压缩包（zipped folders）**形式组织，每个任务对应一个压缩包。每个任务文件夹包含若干条件集文件夹，而每个条件集文件夹下均设有两个子文件夹：
- `input/` — 原始条件集（含元数据与参考图像）
- `model_output/` — 所有纳入模型生成的输出结果

---

### 🐍 **Python 下载方式**
python
from huggingface_hub import snapshot_download
import zipfile
from pathlib import Path

# 下载模型生成结果
local_path = snapshot_download(
repo_id="TIGER-Lab/ImagenWorld-model-outputs",
repo_type="dataset",
local_dir="ImagenWorld-model-outputs",
local_dir_use_symlinks=False,
)

# 解压所有任务压缩包
for zip_file in Path(local_path).glob("*.zip"):
target_dir = Path(local_path) / zip_file.stem
target_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(zip_file, "r") as zf:
zf.extractall(target_dir)
print(f"✅ 已解压 {zip_file.name} → {target_dir}")

---

### 💻 **命令行下载方式**
bash
hf dataset download TIGER-Lab/ImagenWorld-model-outputs --repo-type dataset --local-dir ImagenWorld-model-outputs
cd ImagenWorld-model-outputs && for f in *.zip; do d="${f%.zip}"; mkdir -p "$d"; unzip -q "$f" -d "$d"; done

---

## 📁 数据集结构
解压完成后，你的目录结构将如下所示：

ImagenWorld-model-outputs/
│
├── TIG/
│ ├── TIG_A_000001/
│ │ ├── input/
│ │ │ ├── metadata.json # 任务元数据、提示词与参考图像
│ │ │ ├── 1.png # 参考或条件图像
│ │ │ └── ...
│ │ └── model_output/
│ │ ├── sdxl.png # SDXL 模型生成结果
│ │ ├── gpt-image-1.png # GPT-Image-1 模型生成结果
│ │ ├── gemini.png # Gemini 2.0 Flash 模型生成结果
│ │ └── ...
│ └── ...
│
├── TIE/
├── SRIG/
├── SRIE/
├── MRIG/
└── MRIE/

---

## 🧠 纳入模型
以下为每个ImagenWorld任务所纳入的模型：
- **TIG（文本到图像生成，Text-to-Image Generation）**
SDXL、Infinity、Janus Pro、GPT-Image-1、UNO、BAGEL、Gemini 2.0 Flash、OmniGen 2、Flux.1-Krea-dev、Qwen-Image、Nano Banana
- **TIE（文本与图像编辑，Text + Image Editing）**
InstructPix2Pix、GPT-Image-1、BAGEL、Step1X-Edit、IC-Edit、Gemini 2.0 Flash、OmniGen 2、Flux.1-Kontext-dev、Nano Banana
- **SRIG（单参考图像生成，Single-Reference Image Generation）**
GPT-Image-1、Gemini 2.0 Flash、OmniGen 2、BAGEL、UNO、Nano Banana
- **SRIE（单参考图像编辑，Single-Reference Image Editing）**
GPT-Image-1、Gemini 2.0 Flash、OmniGen 2、BAGEL、Nano Banana
- **MRIG（多参考图像生成，Multi-Reference Image Generation）**
GPT-Image-1、Gemini 2.0 Flash、OmniGen 2、BAGEL、UNO、Nano Banana
- **MRIE（多参考图像编辑，Multi-Reference Image Editing）**
GPT-Image-1、Gemini 2.0 Flash、OmniGen 2、BAGEL、Nano Banana

每个`model_output/`下的文件夹内均存在以对应模型命名的图像文件，例如：

model_output/
├── sdxl.png
├── gpt-image-1.png
├── gemini.png
└── ...

---

## 🧩 任务概览
| 任务缩写 | 任务名称 | 任务描述 |
|------|------|--------------|
| **TIG** | 文本到图像生成（Text-to-Image Generation） | 仅根据文本描述生成图像。 |
| **TIE** | 文本与图像编辑（Text + Image Editing） | 根据文本指令编辑给定图像。 |
| **SRIG** | 单参考图像生成（Single-Reference Image Generation） | 使用单张参考图像与文本提示词生成图像。 |
| **SRIE** | 单参考图像编辑（Single-Reference Image Editing） | 结合文本提示词与单张参考图像编辑图像。 |
| **MRIG** | 多参考图像生成（Multi-Reference Image Generation） | 使用多张参考图像与文本提示生成图像。 |
| **MRIE** | 多参考图像编辑（Multi-Reference Image Editing） | 结合多张参考图像与文本提示编辑图像。 |

---

## 🎨 视觉领域
每个任务涵盖六个视觉领域，以保障跨域鲁棒性：
1. **艺术作品（Artworks）**
2. **写实摄影图像（Photorealistic Images）**
3. **信息图表（Information Graphics）**
4. **文字图形（Textual Graphics）**
5. **计算机图形（Computer Graphics）**
6. **屏幕截图（Screenshots）**

---

## 🔗 相关数据集
| 组件 | 描述 | 仓库链接 |
|------------|--------------|-------------|
| **条件集（Condition Set）** | 输入提示词、元数据与参考图像。 | [`TIGER-Lab/ImagenWorld`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld) |
| **标注集（Annotated Set）** | 包含`train`与`test`两个划分 —— 仅`train`划分包含人工标注；`test`划分仅为剩余未经过人工评估的样本。 | [`TIGER-Lab/ImagenWorld-annotated-set`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-annotated-set) |

---

## 📜 引用说明
若使用ImagenWorld数据集，请引用如下文献：
bibtex
@misc{imagenworld2025,
title = {ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks},
author = {Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Ting Wai Lam and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen},
year = {2025},
doi = {10.5281/zenodo.17344183},
url = {https://zenodo.org/records/17344183},
note = {Community-driven dataset and benchmark release, temporarily archived on Zenodo while arXiv submission is under review.},
}