Ryenhails/ikea-bench

--- license: cc-by-4.0 task_categories: - visual-question-answering - image-text-to-text language: - en tags: - assembly - cross-depiction - vlm-benchmark - ikea - procedural-understanding pretty_name: IKEA-Bench size_categories: - 1K<n<10K --- # IKEA-Bench **Benchmarking Vision-Language Models for Cross-Depiction Assembly Instruction Alignment** [[Project Page]](https://ryenhails.github.io/IKEA-Bench/) | [[Paper]](https://arxiv.org/abs/2604.00913) | [[GitHub]](https://github.com/Ryenhails/IKEA-Bench) ## Dataset Description IKEA-Bench evaluates how well VLMs can align assembly instruction diagrams (like IKEA manuals) with real-world assembly videos. The benchmark contains **1,623 questions** across **6 task types** covering cross-modal alignment and procedural reasoning. This dataset is **self-contained**: all images (133 manual diagrams + 2,570 video frames) are included. No additional downloads needed. ## Dataset Structure ``` ikea-bench/ ├── README.md ├── qa_benchmark.json # 1,623 benchmark questions ├── step_descriptions.json # 132 text descriptions of assembly steps ├── manual_img/ # 133 assembly instruction diagrams │ ├── Bench/{product}/step_{i}/step_{j}.png │ ├── Chair/{product}/... │ ├── Desk/{product}/... │ ├── Misc/{product}/... │ ├── Shelf/{product}/... │ └── Table/{product}/... └── qa_frames/ # 2,570 video frames ├── Bench/{product}/step{i}/{video_id}/frame_*.jpg ├── Chair/{product}/... └── ... ``` ### Question Schema All image paths in `qa_benchmark.json` are **relative to the dataset root**. ```json { "id": "1a_tjusig_step3_MNGqJ4gXqbA_0", "type": "1a", "dimension": "cross_modal", "task": "step_recognition", "product": "tjusig", "category": "Bench", "question": "Which manual step is being performed in these video frames?", "video_frames": ["qa_frames/Bench/tjusig/step3/MNGqJ4gXqbA/frame_00_t52.3s.jpg", ...], "options": [ {"label": "A", "image": "manual_img/Bench/tjusig/step_2/step_5.png", "step_id": 2}, {"label": "B", "image": "manual_img/Bench/tjusig/step_3/step_7.png", "step_id": 3}, ... ], "answer": "B", "answer_step_id": 3, "visual_tokens_est": 4480, "metadata": {...} } ``` ### Task Types | Code | Task | Type | Questions | |------|------|------|-----------| | 1a | Step Recognition | 4-way MC | 320 | | 1b | Action Verification | Binary | 350 | | 2a | Progress Tracking | 4-way MC | 334 | | 2b | Next-Step Prediction | 4-way MC | 204 | | 1c | Video Discrimination | Binary | 350 | | 2c | Sequence Ordering | 4-way MC | 65 | ### Alignment Strategies - **Visual (baseline)**: Video frames + diagram images - **Visual+Text**: Video frames + diagram images + text descriptions - **Text Only**: Video frames + text descriptions (no diagram images) ## Quick Start ### Download the entire dataset ```python from huggingface_hub import snapshot_download # Downloads everything (~300MB) to a local directory snapshot_download( repo_id="Ryenhails/ikea-bench", repo_type="dataset", local_dir="data" ) ``` ### Load and iterate ```python import json from pathlib import Path from PIL import Image data_dir = Path("data") with open(data_dir / "qa_benchmark.json") as f: questions = json.load(f) # Example: load a question with images q = questions[0] video_frames = [Image.open(data_dir / p) for p in q["video_frames"]] option_images = [Image.open(data_dir / o["image"]) for o in q["options"] if "image" in o] print(f"Question: {q['question']}") print(f"Answer: {q['answer']}") print(f"Video frames: {len(video_frames)}, Option images: {len(option_images)}") ``` ### Use with evaluation code ```bash git clone https://github.com/Ryenhails/IKEA-Bench.git cd IKEA-Bench pip install -r requirements.txt # Download data python -c "from huggingface_hub import snapshot_download; snapshot_download('Ryenhails/ikea-bench', repo_type='dataset', local_dir='data')" # Run evaluation python -m ikea_bench.eval \ --model qwen3-vl-8b \ --setting baseline \ --input data/qa_benchmark.json \ --output results/qwen3-vl-8b_baseline.json ``` ## Source Data This benchmark is built upon the [IKEA Manuals at Work](https://github.com/yunongLiu1/IKEA-Manuals-at-Work) dataset (Liu et al., NeurIPS 2024), which provides: - 36 furniture products from 6 categories (29 used in this benchmark) - 98 assembly videos with temporal step annotations - Wordless assembly instruction diagrams Manual diagrams are sourced from the original dataset (CC-BY-4.0). Video frames are extracted from assembly videos hosted on the [Stanford Digital Repository](https://purl.stanford.edu/sg200ps4374). Text descriptions (132 entries) are generated by Claude Opus 4.6 and cross-validated against ground-truth annotations (96.2% consistency). ## Citation ```bibtex @article{liu2026ikeabench, title={Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment}, author={Liu, Zhuchenyang and Zhang, Yao and Xiao, Yu}, journal={arXiv preprint arXiv:2604.00913}, year={2026} } ``` Please also cite the source dataset: ```bibtex @inproceedings{liu2024ikeamanualsatwork, title={IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos}, author={Liu, Yunong and Eyzaguirre, Cristobal and Li, Manling and Khanna, Shubh and Niebles, Juan Carlos and Ravi, Vineeth and Mishra, Saumitra and Liu, Weiyu and Wu, Jiajun}, booktitle={NeurIPS 2024 Datasets and Benchmarks}, year={2024} } ``` ## License CC-BY-4.0. Original IKEA manual images remain the copyright of Inter IKEA Systems B.V. The source dataset [IKEA Manuals at Work](https://github.com/yunongLiu1/IKEA-Manuals-at-Work) is also released under CC-BY-4.0.--- 许可证：CC-BY-4.0 任务类别： - 视觉问答（visual-question-answering） - 图像文本转文本（image-text-to-text） 语言： - 英语（en） 标签： - 装配（assembly） - 跨表征（cross-depiction） - 视觉语言模型基准（vlm-benchmark） - 宜家（ikea） - 过程性理解（procedural-understanding） 展示名称：IKEA-Bench 规模类别： 1K<n<10K --- # IKEA-Bench **面向跨表征装配指令对齐的视觉语言模型基准测试** [[项目页面]](https://ryenhails.github.io/IKEA-Bench/) | [[论文]](https://arxiv.org/abs/2604.00913) | [[GitHub]](https://github.com/Ryenhails/IKEA-Bench) ## 数据集描述 IKEA-Bench用于评估视觉语言模型（Vision-Language Models, VLM）能否将装配指令示意图（如宜家装配手册）与真实世界装配视频对齐。本基准包含**1623个问题**，覆盖**6类任务**，涵盖跨模态对齐（cross-modal alignment）与过程性推理（procedural reasoning）。 本数据集为自包含（self-contained）数据集：包含全部图像（133张手册示意图 + 2570个视频帧），无需额外下载获取外部资源。 ## 数据集结构 ikea-bench/ ├── README.md ├── qa_benchmark.json # 1623个基准问题 ├── step_descriptions.json # 132条装配步骤文本描述 ├── manual_img/ # 133张装配指令示意图 │ ├── Bench/{product}/step_{i}/step_{j}.png │ ├── Chair/{product}/... │ ├── Desk/{product}/... │ ├── Misc/{product}/... │ ├── Shelf/{product}/... │ └── Table/{product}/... └── qa_frames/ # 2570个视频帧 ├── Bench/{product}/step{i}/{video_id}/frame_*.jpg ├── Chair/{product}/... └── ... ### 问题格式 所有在`qa_benchmark.json`中的图像路径均**相对于数据集根目录**。 json { "id": "1a_tjusig_step3_MNGqJ4gXqbA_0", "type": "1a", "dimension": "跨模态（cross_modal）", "task": "步骤识别（step_recognition）", "product": "tjusig", "category": "Bench", "question": "上述视频帧中正在执行的是手册中的哪个装配步骤？", "video_frames": ["qa_frames/Bench/tjusig/step3/MNGqJ4gXqbA/frame_00_t52.3s.jpg", ...], "options": [ {"label": "A", "image": "manual_img/Bench/tjusig/step_2/step_5.png", "step_id": 2}, {"label": "B", "image": "manual_img/Bench/tjusig/step_3/step_7.png", "step_id": 3}, ... ], "answer": "B", "answer_step_id": 3, "visual_tokens_est": 4480, "metadata": {...} } ### 任务类型 | 代码 | 任务名称 | 任务类型 | 问题数量 | |------|----------|----------|----------| | 1a | 步骤识别 | 4路多选 | 320 | | 1b | 动作验证 | 二元分类 | 350 | | 2a | 进度追踪 | 4路多选 | 334 | | 2b | 下一步预测 | 4路多选 | 204 | | 1c | 视频判别 | 二元分类 | 350 | | 2c | 序列排序 | 4路多选 | 65 | ### 对齐策略 - **视觉（基准方法）**：视频帧 + 示意图图像 - **视觉+文本**：视频帧 + 示意图图像 + 文本描述 - **仅文本**：视频帧 + 文本描述（无示意图图像） ## 快速上手 ### 下载完整数据集 python from huggingface_hub import snapshot_download # 下载全部内容（约300MB）至本地目录 snapshot_download( repo_id="Ryenhails/ikea-bench", repo_type="dataset", local_dir="data" ) ### 加载与遍历数据 python import json from pathlib import Path from PIL import Image data_dir = Path("data") with open(data_dir / "qa_benchmark.json") as f: questions = json.load(f) # 示例：加载包含图像的问题 q = questions[0] video_frames = [Image.open(data_dir / p) for p in q["video_frames"]] option_images = [Image.open(data_dir / o["image"]) for o in q["options"] if "image" in o] print(f"问题：{q['question']}") print(f"答案：{q['answer']}") print(f"视频帧数量：{len(video_frames)}，选项图像数量：{len(option_images)}") ### 结合评估代码使用 bash git clone https://github.com/Ryenhails/IKEA-Bench.git cd IKEA-Bench pip install -r requirements.txt # 下载数据集 python -c "from huggingface_hub import snapshot_download; snapshot_download('Ryenhails/ikea-bench', repo_type='dataset', local_dir='data')" # 运行评估 python -m ikea_bench.eval --model qwen3-vl-8b --setting baseline --input data/qa_benchmark.json --output results/qwen3-vl-8b_baseline.json ## 源数据 本基准基于[IKEA Manuals at Work](https://github.com/yunongLiu1/IKEA-Manuals-at-Work)数据集（Liu等人，NeurIPS 2024）构建，该数据集提供： - 6个品类下的36件家具产品（本基准使用其中29件） - 98个带有时间步标注的装配视频 - 无文字的装配指令示意图 示意图源自原数据集（CC-BY-4.0许可）。视频帧源自[斯坦福数字知识库](https://purl.stanford.edu/sg200ps4374)中托管的装配视频。 文本描述（共132条）由Claude Opus 4.6生成，并与真实标注进行交叉验证（一致性达96.2%）。 ## 引用 bibtex @article{liu2026ikeabench, title={Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment}, author={Liu, Zhuchenyang and Zhang, Yao and Xiao, Yu}, journal={arXiv preprint arXiv:2604.00913}, year={2026} } 请同时引用源数据集： bibtex @inproceedings{liu2024ikeamanualsatwork, title={IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos}, author={Liu, Yunong and Eyzaguirre, Cristobal and Li, Manling and Khanna, Shubh and Niebles, Juan Carlos and Ravi, Vineeth and Mishra, Saumitra and Liu, Weiyu and Wu, Jiajun}, booktitle={NeurIPS 2024 Datasets and Benchmarks}, year={2024} } ## 许可 本数据集采用CC-BY-4.0许可。原始宜家手册图像的版权归Inter IKEA Systems B.V.所有。源数据集[IKEA Manuals at Work](https://github.com/yunongLiu1/IKEA-Manuals-at-Work)同样采用CC-BY-4.0许可发布。