KRAFTON/VLM-SubtleBench

--- license: cc-by-nc-4.0 task_categories: - visual-question-answering - image-to-text language: - en tags: - vlm - benchmark - comparative-reasoning - subtle-difference - image-comparison - multi-image size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: test path: data/test.jsonl - split: val path: data/val.jsonl dataset_info: features: - name: image_1 dtype: image - name: image_2 dtype: image - name: question dtype: string - name: answer dtype: string - name: distractors sequence: string - name: has_caption dtype: bool - name: caption dtype: string - name: category dtype: string - name: domain dtype: string - name: source dtype: string - name: source_id dtype: string - name: raw_folder dtype: string - name: generation_info dtype: string --- # VLM-SubtleBench **VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?** The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imaging, and aerial surveillance. While comparative reasoning benchmarks for vision-language models (VLMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. VLM-SubtleBench is a benchmark designed to evaluate VLMs on **subtle comparative reasoning** — detecting fine-grained differences between highly similar image pairs that are easy for humans but challenging for state-of-the-art VLMs. Unlike prior benchmarks restricted to natural image datasets, VLM-SubtleBench spans diverse domains including industrial, aerial, and medical imagery. ## Benchmark Summary | | | |---|---| | **Total QA pairs** | 12,923 | | **Difference types** | 10 | | **Image domains** | 6 (Natural, Industrial, Aerial, Synthetic, Medical) | | **Data sources** | 14 | | **Human captions** | 1,200 | | **Splits** | test (11,688) / val (1,235) | | **Task format** | Multiple-choice VQA + Image Difference Captioning | > **Note**: Medical domain images (MIMIC-CXR, 362 pairs) are not included due to licensing restrictions, but their QA entries are included in `qa.json`. See [Medical Data](#medical-data-mimic-cxr) below for instructions on how to obtain the images. ## Medical Data (MIMIC-CXR) The medical domain QA entries (362 attribute comparison pairs from MIMIC-CXR chest X-rays, 664 unique images) are included in `qa.json`, but the corresponding images are not included due to [PhysioNet licensing requirements](https://physionet.org/content/mimic-cxr-jpg/2.1.0/). ### Step 1: Obtain PhysioNet Credentialed Access 1. Create an account at [PhysioNet](https://physionet.org/) 2. Complete the required [CITI training course](https://physionet.org/about/citi-course/) for "Data or Specimens Only Research" 3. Go to [MIMIC-CXR-JPG v2.1.0](https://physionet.org/content/mimic-cxr-jpg/2.1.0/) and sign the data use agreement 4. Wait for your access to be approved ### Step 2: Download Images We provide a script that automatically downloads only the 664 images required by `qa.json` and places them at the expected paths (`images/mimic/...`). ```bash python download_mimic.py --user <physionet-username> --password <physionet-password> ``` The script: - Parses `qa.json` to find all required MIMIC-CXR image paths - Downloads each image from PhysioNet via `wget` - Places them under `images/mimic/` preserving the original directory hierarchy (e.g., `images/mimic/p15/p15592981/s55194630/{hash}.jpg`) - Skips images that already exist, so it is safe to re-run You can also download individual images manually: ```bash wget --user <username> --password <password> \ https://physionet.org/files/mimic-cxr-jpg/2.1.0/files/p15/p15000170/s54385701/3ea0cd5d-b6ef4a9d-bd053deb-a611067c-284e4144.jpg \ -O images/mimic/p15/p15000170/s54385701/3ea0cd5d-b6ef4a9d-bd053deb-a611067c-284e4144.jpg ``` ## Download and Evaluation ### Download ```bash # Using huggingface_hub pip install huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download('KRAFTON/VLM-SubtleBench', repo_type='dataset', local_dir='VLM-SubtleBench')" ``` Or clone directly with Git LFS: ```bash git lfs install git clone https://huggingface.co/datasets/KRAFTON/VLM-SubtleBench ``` ### Evaluation For evaluation code and instructions, please refer to the official GitHub repository: https://github.com/krafton-ai/VLM-SubtleBench ## Citation ```bibtex @inproceedings{kim2026vlmsubtlebench, title={VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?}, author={Kim, Minkyu and Lee, Sangheon and Park, Dongmin}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026}, url={https://arxiv.org/abs/2603.07888} } ```许可证：CC-BY-NC-4.0 任务类别： - 视觉问答（Visual Question Answering） - 图像到文本 语言： - 英语 标签： - VLM - 基准测试集 - 比较推理 - 细微差异 - 图像对比 - 多图像 样本规模类别：10000 < n < 100000 配置项： - 配置名称：default 数据文件： - 拆分方式：测试集，路径：data/test.jsonl - 拆分方式：验证集，路径：data/val.jsonl 数据集信息： 特征字段： - image_1：图像类型 - image_2：图像类型 - question：字符串类型 - answer：字符串类型 - distractors：字符串序列类型 - has_caption：布尔类型 - caption：字符串类型 - category：字符串类型 - domain：字符串类型 - source：字符串类型 - source_id：字符串类型 - raw_folder：字符串类型 - generation_info：字符串类型 --- # VLM-SubtleBench：探究视觉语言模型与人类级细微比较推理能力的差距 **VLM-SubtleBench：视觉语言模型距离人类级细微比较推理能力尚有几何？** 区分视觉相似图像间细微差异的能力，在工业异常检测、医学影像、航空监视等诸多领域均至关重要。尽管针对视觉语言模型（Vision-Language Model, VLM）的比较推理基准近期相继涌现，但此类基准大多聚焦于差异显著的图像，无法捕捉真实应用场景中所需的精细化推理能力。 VLM-SubtleBench是一款专为评估视觉语言模型**细微比较推理能力**而设计的基准测试集，其任务为检测高度相似图像对之间的细粒度差异——此类差异对人类而言轻而易举，但对当前顶尖的视觉语言模型却极具挑战性。与此前仅局限于自然图像数据集的基准不同，VLM-SubtleBench覆盖了工业、航空、医学影像等多元领域。 ## 基准测试集概览 | | | |---|---| | **总问答对数量** | 12,923 | | **差异类型数量** | 10 | | **图像领域数量** | 6（自然图像、工业图像、航空图像、合成图像、医学图像） | | **数据来源数量** |14| | **人工标注字幕数量** |1,200| | **数据集拆分** |测试集（11,688）/验证集（1,235）| | **任务格式** | 多选视觉问答 + 图像差异字幕生成 | > **注意**：受许可证限制，医学领域图像（MIMIC-CXR，共362对）未包含在数据集中，但其对应的问答条目已收录于`qa.json`。如需获取图像的相关说明，请参见下文的[医学数据（MIMIC-CXR）](#medical-data-mimic-cxr)章节。 ## 医学数据（MIMIC-CXR） 医学领域的问答条目（源自MIMIC-CXR胸部X光片的362对属性比较对，涉及664张唯一图像）已收录于`qa.json`，但由于[PhysioNet许可证要求](https://physionet.org/content/mimic-cxr-jpg/2.1.0/)，对应的图像未包含在数据集中。 ### 步骤1：获取PhysioNet授权访问权限 1. 在[PhysioNet官网](https://physionet.org/)注册账号 2. 完成针对“仅数据或样本研究”的[CITI培训课程](https://physionet.org/about/citi-course/) 3. 访问[MIMIC-CXR-JPG v2.1.0](https://physionet.org/content/mimic-cxr-jpg/2.1.0/)并签署数据使用协议 4. 等待访问权限获批 ### 步骤2：下载图像 我们提供了一款脚本，可自动仅下载`qa.json`所需的664张图像，并将其放置至指定路径（`images/mimic/...`）。 bash python download_mimic.py --user <physionet-username> --password <physionet-password> 该脚本的功能如下： - 解析`qa.json`以获取所有所需的MIMIC-CXR图像路径 - 通过`wget`从PhysioNet下载每张图像 - 将图像保存至`images/mimic/`目录下，并保留原始目录结构（例如`images/mimic/p15/p15592981/s55194630/{hash}.jpg`） - 跳过已存在的图像，因此可安全重复运行 您也可以手动下载单张图像： bash wget --user <username> --password <password> https://physionet.org/files/mimic-cxr-jpg/2.1.0/files/p15/p15000170/s54385701/3ea0cd5d-b6ef4a9d-bd053deb-a611067c-284e4144.jpg -O images/mimic/p15/p15000170/s54385701/3ea0cd5d-b6ef4a9d-bd053deb-a611067c-284e4144.jpg ## 下载与评估 ### 下载方式 bash # 使用huggingface_hub pip install huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download('KRAFTON/VLM-SubtleBench', repo_type='dataset', local_dir='VLM-SubtleBench')" 或直接通过Git LFS克隆： bash git lfs install git clone https://huggingface.co/datasets/KRAFTON/VLM-SubtleBench ### 评估 如需评估代码与操作说明，请参阅官方GitHub仓库： https://github.com/krafton-ai/VLM-SubtleBench ## 引用 bibtex @inproceedings{kim2026vlmsubtlebench, title={VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?}, author={Kim, Minkyu and Lee, Sangheon and Park, Dongmin}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026}, url={https://arxiv.org/abs/2603.07888} }