ICE-Bench

# ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing

<font size=3><div align='center' > [[🍎 Project Page](https://ali-vilab.github.io/ICE-Bench-Page/)] [[📖 arXiv Paper](https://arxiv.org/abs/2503.14482)] [[🤗 Dataset](https://huggingface.co/datasets/ali-vilab/ICE-Bench)] </div></font>

---

## 🔥 News

* **`2025.8.26`** The code and dataset for automated evaluation are available now.
* **`2025.6.26`** Our paper has been received by ICCV 2025!
* **`2025.3.18`** Paper is available on Arxiv.

## Abstract

<p align="center">
<img src="./assets/teaser.png" height="100%">
</p>

## Evaluation

### 1. Environment Setup

Set up the environment for running the evaluation scripts.

```bash
pip install -r requirements.txt
```

### 2. Download and Prepare the Dataset and Models

Download the evaluation data and models from [Hugging Face repo](https://huggingface.co/datasets/ali-vilab/ICE-Bench).
Then unzip `data.zip` and`models.zip` under the root of ICE-Bench project.

For Qwen2.5-VL-72B-Instruct, you should download it from the [official repo](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) and place it in the `models` folder under the root of this project.

### 3. Run your Model to Generate Results

Run your model to generate the results for all tasks. Save the generated images in the `results/{METHOD_NAME}/images` folder,
and keep an json file that contains (item_id, image_save_path) key-value pairs.

Your directory structure should look like this:

```
ICE-Bench/
├── assets/
├── dataset/
│ ├── images/
│ └── data.jsonl
├── models/
│ ├── Qwen2.5-VL-72B-Instruct
│ ├── aesthetic_predictor_v2_5.pth
│ └── ...
├── results/
│ └── method_name/
│ ├── images/
│ │ ├── image1.jpg
│ │ ├── image2.jpg
│ │ └── ...
│ └── gen_info.json
├── evaluators/
├── config.py
├── requirements.txt
├── cal_scores.py
├── eval.py
└── ...
```

The `gen_info.json` file look like this:

```
{
"item_id1": "results/{METHOD}/images/image1.jpg",
"item_id2": "results/{METHOD}/images/image2.jpg",
...
}
```

### 4. Run Evaluation

```bash
python eval.py -m dataset/data.jsonl -f results/{METHOD}/gen_info.json -s results/{METHOD}/eval_result.txt
```

The evaluation results will be saved in the `results/{METHOD}/eval_result.txt` file.

### 5. Calculate Task Scores and Method Scores

```bash
python cal_scores.py -f results/{METHOD}/eval_result.txt
```

## Citation

If you find our work helpful for your research, please consider citing our work.

```bibtex
@article{pan2025ice,
title={Ice-bench: A unified and comprehensive benchmark for image creating and editing},
author={Pan, Yulin and He, Xiangteng and Mao, Chaojie and Han, Zhen and Jiang, Zeyinzi and Zhang, Jingfeng and Liu, Yu},
journal={arXiv preprint arXiv:2503.14482},
year={2025}
}
```

# ICE-Bench：面向图像生成与编辑的统一综合基准测试

<font size=3><div align='center' > [[🍎 项目主页](https://ali-vilab.github.io/ICE-Bench-Page/)] [[📖 arXiv论文](https://arxiv.org/abs/2503.14482)] [[🤗 数据集](https://huggingface.co/datasets/ali-vilab/ICE-Bench)] </div></font>

---

## 🔥 最新动态

* **`2025.8.26`** 自动化评估的代码与数据集现已公开。
* **`2025.6.26`** 我们的论文已被2025年国际计算机视觉大会（ICCV）收录！
* **`2025.3.18`** 论文已在ArXiv平台发布。

## 摘要

<p align="center">
<img src="./assets/teaser.png" height="100%">
</p>

## 评估

### 1. 环境配置

配置运行评估脚本所需的运行环境。

bash
pip install -r requirements.txt

### 2. 下载并准备数据集与模型

从[Hugging Face仓库](https://huggingface.co/datasets/ali-vilab/ICE-Bench)下载评估数据与模型。随后将`data.zip`与`models.zip`解压至ICE-Bench项目的根目录下。

对于Qwen2.5-VL-72B-Instruct，需从[官方仓库](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)下载，并放置于本项目根目录下的`models`文件夹中。

### 3. 运行模型以生成结果

运行你的模型以完成所有任务的结果生成。将生成的图像保存至`results/{METHOD_NAME}/images`文件夹中，并保留一个包含`(item_id, image_save_path)`键值对的JSON文件。

你的目录结构应如下所示：

ICE-Bench/
├── assets/
├── dataset/
│ ├── images/
│ └── data.jsonl
├── models/
│ ├── Qwen2.5-VL-72B-Instruct
│ ├── aesthetic_predictor_v2_5.pth
│ └── ...
├── results/
│ └── method_name/
│ ├── images/
│ │ ├── image1.jpg
│ │ ├── image2.jpg
│ │ └── ...
│ └── gen_info.json
├── evaluators/
├── config.py
├── requirements.txt
├── cal_scores.py
├── eval.py
└── ...

`gen_info.json`文件格式示例如下：

json
{
"item_id1": "results/{METHOD}/images/image1.jpg",
"item_id2": "results/{METHOD}/images/image2.jpg",
...
}

### 4. 执行评估

bash
python eval.py -m dataset/data.jsonl -f results/{METHOD}/gen_info.json -s results/{METHOD}/eval_result.txt

评估结果将保存至`results/{METHOD}/eval_result.txt`文件中。

### 5. 计算任务得分与方法得分

bash
python cal_scores.py -f results/{METHOD}/eval_result.txt

## 引用

若您的研究工作中使用了我们的成果，请考虑引用本文。

bibtex
@article{pan2025ice,
title={Ice-bench: A unified and comprehensive benchmark for image creating and editing},
author={Pan, Yulin and He, Xiangteng and Mao, Chaojie and Han, Zhen and Jiang, Zeyinzi and Zhang, Jingfeng and Liu, Yu},
journal={arXiv preprint arXiv:2503.14482},
year={2025}
}