kontext-bench

# Kontext Bench

Kontext Bench is a benchmark for image editing models consisting of source images paired with image editing instructions and category tags.

The benchmark comprises 1026 unique image-prompt pairs derived from 108 base images from diverse sources. It spans five core tasks: local instruction editing, global instruction editing, text editing, style reference, and character reference. We found that the scale of the benchmark provides a good balance between reliable human evaluation and comprehensive coverage of real-world applications.

## Benchmark Structure

```
kontext-bench/
└── test/
├── images/
│ ├── 0000.jpg
│ ├── 0001.jpg
│ └── ...
└── metadata.jsonl
```

## Fields

Each line in `metadata.jsonl` consists of

- `file_name`: Path to the image file relative to the split directory
- `instruction`: The editing instruction to apply to the image
- `category`: Category of the editing instruction
- `key`: Unique identifier for this image-instruction pair
- `image_idx`: Index of the source image
- `prompt_idx`: Index of the prompt for this image

An example entry is shown below:

```
{
"file_name": "images/0000.jpg"
"instruction": "give the cat a tophat",
"category": "Instruction Editing - Local",
"key": "0000_01",
"img_idx": "0000",
"prompt_idx": "01"
}
```

## Benchmark Statistics

- Total entries: 1026
- Unique images: 108

## Category Statistics

- Character Reference: 193 entries
- Instruction Editing - Global: 262 entries
- Instruction Editing - Local: 416 entries
- Style Reference: 63 entries
- Text Editing: 92 entries

## License
The benchmark is released under the MIT License. This benchmark and the included Images are made available for scientific and research purposes only. We gratefully acknowledge all contributing photographers, Unsplash, Pexels for making their visuals available to the research community.

## Citation

```bib
@misc{labs2025flux1kontextflowmatching,
title={FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space}, Add commentMore actions
author={Black Forest Labs and Stephen Batifol and Andreas Blattmann and Frederic Boesel and Saksham Consul and Cyril Diagne and Tim Dockhorn and Jack English and Zion English and Patrick Esser and Sumith Kulal and Kyle Lacey and Yam Levi and Cheng Li and Dominik Lorenz and Jonas Müller and Dustin Podell and Robin Rombach and Harry Saini and Axel Sauer and Luke Smith},
year={2025},
eprint={2506.15742},
archivePrefix={arXiv},
primaryClass={cs.GR},
url={https://arxiv.org/abs/2506.15742},
}
```

# Kontext基准测试集（Kontext Bench）
Kontext基准测试集是一款面向图像编辑模型的基准测试集，由源图像、对应的图像编辑指令与类别标签组合而成。

该基准测试集包含1026组独特的图像-指令对，其数据源自108张来自不同渠道的基础图像。测试集涵盖五大核心任务：局部指令编辑（local instruction editing）、全局指令编辑（global instruction editing）、文本编辑（text editing）、风格参考（style reference）与角色参考（character reference）。经评估，该基准测试集的规模能够在可靠的人工评估需求与真实应用场景的全面覆盖之间实现良好平衡。

## 基准测试集结构

kontext-bench/
└── test/
├── images/
│ ├── 0000.jpg
│ ├── 0001.jpg
│ └── ...
└── metadata.jsonl

## 字段说明
metadata.jsonl 文件中的每一行包含以下字段：
- `file_name`：图像文件相对于拆分目录的路径
- `instruction`：应用于该图像的编辑指令
- `category`：编辑指令的类别
- `key`：该图像-指令对的唯一标识符
- `image_idx`：源图像的索引
- `prompt_idx`：该图像对应指令的索引

示例条目如下：

{
"file_name": "images/0000.jpg",
"instruction": "give the cat a tophat",
"category": "Instruction Editing - Local",
"key": "0000_01",
"img_idx": "0000",
"prompt_idx": "01"
}

## 基准测试集统计信息
- 总条目数：1026
- 唯一源图像数：108

## 类别统计信息
- 角色参考（character reference）：193条
- 全局指令编辑（global instruction editing）：262条
- 局部指令编辑（local instruction editing）：416条
- 风格参考（style reference）：63条
- 文本编辑（text editing）：92条

## 许可证
本基准测试集采用MIT许可证发布。本基准测试集及其中包含的图像仅可用于科学研究用途。我们衷心感谢所有供稿摄影师，以及Unsplash、Pexels平台为研究社区开放其视觉素材。

## 引用格式
bib
@misc{labs2025flux1kontextflowmatching,
title={FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space}, Add commentMore actions
author={Black Forest Labs and Stephen Batifol and Andreas Blattmann and Frederic Boesel and Saksham Consul and Cyril Diagne and Tim Dockhorn and Jack English and Zion English and Patrick Esser and Sumith Kulal and Kyle Lacey and Yam Levi and Cheng Li and Dominik Lorenz and Jonas Müller and Dustin Podell and Robin Rombach and Harry Saini and Axel Sauer and Luke Smith},
year={2025},
eprint={2506.15742},
archivePrefix={arXiv},
primaryClass={cs.GR},
url={https://arxiv.org/abs/2506.15742},
}