Beeemo/ref-adv-s
收藏资源简介：
--- license: cc-by-4.0 task_categories: - visual-question-answering - object-detection language: - en tags: - referring-expression-comprehension - visual-grounding - mllm - benchmark size_categories: - 1K<n<10K --- # Ref-Adv-s 🏠[Website](https://ref-adv.github.io) | 🖥️[Code](https://github.com/dddraxxx/Ref-Adv) | 📊[Results](https://ref-adv.github.io/#results) | 📄[Paper](https://arxiv.org/abs/2602.23898) **Ref-Adv-s** is the publicly released subset of the Ref-Adv benchmark from our paper **"Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks"** (ICLR 2026). ## Overview **Referring Expression Comprehension (REC)** links natural language to region-level visual perception — given an image and a text expression, the task is to localize the described object. Standard benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg have driven years of progress, yet they harbor critical shortcuts: 1. **Expressions are too short** (avg. ~3 words), leaving little reasoning demand. 2. **Few visual distractors** make the target easy to find by elimination. 3. **Redundant descriptors** let models latch onto a single cue and ignore the rest. <p align="center"> <img src="https://ref-adv.github.io/static/images/teaser.png" width="85%"> </p> **Ref-Adv-s** is a publicly released subset of **1,142 curated cases** from the Ref-Adv benchmark, a modern REC benchmark designed to suppress these shortcuts by pairing complex referring expressions with hard visual distractors. Images are sourced from **COCO val2017** and **OpenImages**. ## Dataset Schema | Column | Type | Description | |---|---|---| | `image` | Image | The input image | | `file_name` | string | Original image filename | | `image_file` | string | Image filename in this dataset | | `normal_caption` | string | Referring expression describing the target object | | `problem` | string | Full prompt with the referring expression | | `solution` | list[float] | Ground-truth bounding box in absolute `[x1, y1, x2, y2]` format | | `normalized_solution` | list[int] | Ground-truth bounding box normalized to 1000-scale `[x1, y1, x2, y2]` | | `width` | int | Image width in pixels | | `height` | int | Image height in pixels | | `image_source` | string | Source dataset (`coco_val2017` or `openimages`) | | `human_authored` | bool | Whether the caption is human-written | | `use_negation` | bool | Whether the caption uses negation | | `distractors` | string | Number of distractor objects in the image | | `row_idx` | int | Row index | ## Usage ```python from datasets import load_dataset ds = load_dataset("dddraxxx/ref-adv-s", split="train") print(ds[0]) ``` ## Evaluation See our [evaluation code](https://github.com/dddraxxx/Ref-Adv) for running model inference and computing metrics (Acc@0.5, Acc@0.75, Acc@0.9, distractor-bin breakdowns). Our [results](https://github.com/dddraxxx/Ref-Adv/tree/main/outputs/qwen) for all Qwen 2.5–3.5 VL models are also available. ## Contact For questions or issues, you may contact Qihua Dong via email [dongqh078@gmail.com](mailto:dongqh078@gmail.com). ## Citation ```bibtex @article{dong2026refadv, title = {Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks}, author = {Qihua Dong and Kuo Yang and Lin Ju and Handong Zhao and Yitian Zhang and Yizhou Wang and Huimin Zeng and Jianglin Lu and Yun Fu}, year = {2026}, journal = {arXiv preprint arXiv: 2602.23898} } ``` ## License This dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The copyright of the images belongs to the original sources (COCO, OpenImages).
--- 许可证：CC BY 4.0 任务类别： - 视觉问答 - 目标检测 语言： - 英语 标签： - 指代理解（Referring Expression Comprehension） - 视觉接地（Visual Grounding） - 多模态大语言模型（Multimodal Large Language Model, MLLM） - 基准测试数据集 样本规模：1K<n<10K --- # Ref-Adv-s 🏠[官网](https://ref-adv.github.io) | 🖥️[代码](https://github.com/dddraxxx/Ref-Adv) | 📊[评测结果](https://ref-adv.github.io/#results) | 📄[论文](https://arxiv.org/abs/2602.23898) **Ref-Adv-s** 是我们发表于ICLR 2026的论文《Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks》中公开的Ref-Adv基准测试子集。 ## 数据集概览 **指代理解（Referring Expression Comprehension, REC）** 旨在建立自然语言与区域级视觉感知的关联——给定一张图像与一段文本描述，任务目标为定位描述中的目标对象。RefCOCO、RefCOCO+、RefCOCOg等经典基准数据集推动了该领域多年发展，但存在若干关键缺陷： 1. **指代表达式过短**（平均约3个词），几乎无需复杂推理即可完成任务； 2. **视觉干扰物过少**，模型可通过排除法轻松定位目标； 3. **描述存在冗余信息**，模型仅需捕捉单个线索即可完成任务，无需理解完整描述。 <p align="center"> <img src="https://ref-adv.github.io/static/images/teaser.png" width="85%"> </p> **Ref-Adv-s** 是Ref-Adv基准测试中公开的1142个精选样本子集，该基准为新型REC基准数据集，通过将复杂指代表达式与高难度视觉干扰物配对，有效规避了上述缺陷。图像数据源自**COCO val2017**与**OpenImages**。 ## 数据集结构 | 列名 | 数据类型 | 描述 | |---|---|---| | `image` | 图像 | 输入图像 | | `file_name` | 字符串 | 原始图像文件名 | | `image_file` | 字符串 | 本数据集中的图像文件名 | | `normal_caption` | 字符串 | 描述目标对象的指代表达式 | | `problem` | 字符串 | 包含指代表达式的完整提示词 | | `solution` | 浮点数组 | 采用绝对坐标格式`[x1, y1, x2, y2]`的真实边界框 | | `normalized_solution` | 整数数组 | 归一化至1000尺度的真实边界框`[x1, y1, x2, y2]` | | `width` | 整数 | 图像宽度（像素） | | `height` | 整数 | 图像高度（像素） | | `image_source` | 字符串 | 图像来源数据集（`coco_val2017`或`openimages`） | | `human_authored` | 布尔值 | 该描述是否由人工撰写 | | `use_negation` | 布尔值 | 该描述是否包含否定表述 | | `distractors` | 字符串 | 图像中干扰物的数量 | | `row_idx` | 整数 | 行索引 | ## 使用方法 python from datasets import load_dataset ds = load_dataset("dddraxxx/ref-adv-s", split="train") print(ds[0]) ## 评测方法 可参考我们的[评测代码](https://github.com/dddraxxx/Ref-Adv)运行模型推理并计算评测指标（Acc@0.5、Acc@0.75、Acc@0.9以及干扰物分箱细分结果）。我们还公开了所有Qwen 2.5–3.5 VL模型的[评测结果](https://github.com/dddraxxx/Ref-Adv/tree/main/outputs/qwen)。 ## 联系方式 若有疑问或问题，可通过邮箱[dongqh078@gmail.com](mailto:dongqh078@gmail.com)联系董启华。 ## 引用格式 bibtex @article{dong2026refadv, title = {Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks}, author = {Qihua Dong and Kuo Yang and Lin Ju and Handong Zhao and Yitian Zhang and Yizhou Wang and Huimin Zeng and Jianglin Lu and Yun Fu}, year = {2026}, journal = {arXiv preprint arXiv: 2602.23898} } ## 许可协议 本数据集采用[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)许可协议。图像的版权归原始来源（COCO、OpenImages）所有。
