Arena-T2I-Hard

# Arena-T2I-Hard A **310-prompt stress benchmark** for evaluating **faithfulness** (prompt-following) of text-to-image models, drawn from real, hard arena user requests — long, multi-entity prompts with attributes, spatial relations, counts, and stylistic constraints. Each prompt ships **pre-decomposed into a dependency-aware DAG of yes/no questions**; when scoring an image, failing a parent question zeroes out its descendants. The benchmark stays discriminative where DPG-Bench and DSG saturate. - 🌐 Project page: https://banyuanhao.github.io/Arena-T2I-Hard-Page/ - 💻 Evaluation code: https://github.com/banyuanhao/Arena-T2I-Hard - 📄 Paper: https://arxiv.org/abs/2606.31711 ## Files - `benchmark.json` — a JSON array of 310 records. ## Record schema ```jsonc { "prompt_id": 0, // 0..309 "prompt": "Draw me an image ...", "questions": [ { "id": 0, "question": "Is there a warrior in the image?", "depends_on": [], // parent question ids (empty = root) "type": "faithfulness" // "faithfulness" or "aesthetics" }, { "id": 1, "question": "Is the warrior holding arrows?", "depends_on": [0], "type": "faithfulness" } // ... ] } ``` Across the 310 prompts there are ~13.9k questions (~9.6k `faithfulness`, ~4.4k `aesthetics`). The paper's faithfulness metric uses faithfulness questions only. ## Usage ```python from datasets import load_dataset ds = load_dataset("lmarena-ai/Arena-T2I-Hard", split="test") print(ds[0]["prompt"]) print(ds[0]["questions"][0]) # {'id': 0, 'question': '...', 'depends_on': [], 'type': 'faithfulness'} ``` Or grab the raw JSON file directly: ```python import json from huggingface_hub import hf_hub_download path = hf_hub_download("lmarena-ai/Arena-T2I-Hard", "benchmark.json", repo_type="dataset") data = json.load(open(path)) ``` ## Scoring To score a text-to-image model on this benchmark (generate images for the 310 prompts, then run the dependency-aware checklist judge), see the evaluation code at https://github.com/banyuanhao/Arena-T2I-Hard. ## License MIT. ## Citation ```bibtex @misc{ban2026arenat2ihard, title = {Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist}, author = {Ban, Yuanhao and Xie, Tong and An, Sohyun and Hong, Yunqi and Frick, Evan and Hsu, I-Hung and Chiang, Wei-Lin and Stoica, Ion and Hsieh, Cho-Jui}, year = {2026}, eprint = {2606.31711}, archivePrefix = {arXiv}, primaryClass = {cs.AI}, url = {https://arxiv.org/abs/2606.31711} } ```