uclanlp/MRAG-Bench

--- language: - en license: cc-by-4.0 size_categories: - 1K<n<10K task_categories: - question-answering - visual-question-answering - multiple-choice pretty_name: MRAG-Bench dataset_info: features: - name: id dtype: string - name: aspect dtype: string - name: scenario dtype: string - name: image dtype: image - name: gt_images sequence: image - name: question dtype: string - name: A dtype: string - name: B dtype: string - name: C dtype: string - name: D dtype: string - name: answer_choice dtype: string - name: answer dtype: string - name: image_type dtype: string - name: source dtype: string - name: retrieved_images sequence: image splits: - name: test num_bytes: 4740745536 num_examples: 1353 configs: - config_name: default data_files: - split: test path: data/test-* --- # MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models [**🌐 Homepage**](https://mragbench.github.io/) | [**📖 Paper**](https://arxiv.org/abs/2410.08182) | [**💻 Evaluation**](https://github.com/mragbench/MRAG-Bench) ## Intro MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities. <img src="https://gordonhu608.github.io/images/mragbench_teaser.png" width="1000" /> ## Results Evaluated upon 10 open-source and 4 proprietary LVLMs, our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively. <img src="https://gordonhu608.github.io/images/mragbench_qual.png" width="800" /> ## Load Dataset The `data/` directory contains the full dataset annotations and images pre-loaded for processing with HF Datasets. It can be loaded as follows: ```python from datasets import load_dataset mrag_bench = load_dataset("uclanlp/MRAG-Bench", split="test") ``` ## Dataset Description The dataset contains the following fields: | Field Name | Description | | :--------- | :---------- | | `id` | Unique identifier for the example | | `aspect`| Aspect type for the example | | `scenario` | The type of scenario associated with the entry | | `image`| Contains image data in byte format | | `gt_images`| A list of top 5 ground-truth images information | | `question` | Question asked about the image | | `A` | Choice A for the question | | `B` | Choice B for the question | | `C` | Choice C for the question | | `D` | Choice D for the question | |`answer_choice`| Correct choice identifier | | `answer` | Correct answer to the question | | `image_type`| Type of image object | | `source`| Source of the image | | `retrieved_images`| A list of top 5 retrieved images information by CLIP | <br> We release the image corpus [here](https://drive.google.com/file/d/1atwkNXH3aEtCLuqimZoB1Mifj5CwL3CL/view?usp=sharing) for retrieval. <br> ## Contact * Wenbo Hu: [e-mail] ## Citation ``` @article{hu2024mragbench, title={MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models}, author={Hu, Wenbo and Gu, Jia-Chen and Dou, Zi-Yi and Fayyaz, Mohsen and Lu, Pan and Chang, Kai-Wei and Peng, Nanyun}, journal={arXiv preprint arXiv:2410.08182}, year={2024} } ```# MRAG-Bench：面向检索增强多模态模型的视觉导向评估 [🌐 项目主页](https://mragbench.github.io/) | [📖 学术论文](https://arxiv.org/abs/2410.08182) | [💻 评估代码库](https://github.com/mragbench/MRAG-Bench) ## 数据集元信息 - 语言：英语 - 许可协议：CC BY 4.0 - 样本规模类别：1000 < 样本数 < 10000 - 任务类别：问答、视觉问答、多项选择 - 美观名称：MRAG-Bench ### 数据集详情 #### 特征字段 1. `id`：字符串类型，样本唯一标识符 2. `aspect`：字符串类型，样本维度类型 3. `scenario`：字符串类型，样本关联场景类型 4. `image`：图像类型，存储图像字节数据 5. `gt_images`：图像序列类型，包含前5张基准真值（ground-truth）图像 6. `question`：字符串类型，针对图像的问题文本 7. `A`：字符串类型，试题选项A 8. `B`：字符串类型，试题选项B 9. `C`：字符串类型，试题选项C 10. `D`：字符串类型，试题选项D 11. `answer_choice`：字符串类型，正确选项标识符 12. `answer`：字符串类型，试题正确答案 13. `image_type`：字符串类型，图像类型 14. `source`：字符串类型，图像来源 15. `retrieved_images`：图像序列类型，由CLIP检索得到的前5张候选图像 #### 数据划分 - 测试集（test）：数据量4740745536字节，共1353个样本 #### 配置信息 - 默认配置：数据文件路径为`data/test-*`，对应测试集划分 ## 简介 MRAG-Bench 包含16130张图像与1353道经人工标注的多项选择题，覆盖9类不同场景，旨在为大视觉语言模型（Large Vision Language Model, LVLM）的视觉导向多模态检索增强生成（Retrieval-Augmented Generation, RAG）能力提供全面且系统的评估。  ## 评估结果 我们在10个开源与4个闭源大视觉语言模型上开展了评估，结果显示，相较于仅补充文本知识，所有模型在补充图像信息后均获得了更显著的性能提升。值得注意的是，性能最优的模型GPT-4o在有效利用检索得到的知识方面仍存在瓶颈：其仅依靠基准真值图像仅获得5.82%的性能提升，而人类参与者借助同类信息可实现33.16%的性能提升。该结果凸显了MRAG-Bench的重要价值，可推动学界进一步提升大视觉语言模型对检索到的视觉知识的利用能力。  ## 数据集加载 `data/` 目录包含完整的数据集标注与预加载的图像文件，可通过Hugging Face Datasets库直接加载，加载代码如下： python from datasets import load_dataset mrag_bench = load_dataset("uclanlp/MRAG-Bench", split="test") ## 数据集字段说明 下表列出了数据集各字段的含义： | 字段名 | 描述 | | :--------- | :---------- | | `id` | 样本的唯一标识符 | | `aspect` | 样本所属的维度类型 | | `scenario` | 样本关联的场景类型 | | `image` | 以字节格式存储的图像数据 | | `gt_images` | 包含前5张基准真值（ground-truth）图像的序列 | | `question` | 针对图像提出的问题文本 | | `A` | 试题选项A | | `B` | 试题选项B | | `C` | 试题选项C | | `D` | 试题选项D | |`answer_choice`| 正确选项的标识符 | | `answer` | 试题的正确答案文本 | | `image_type`| 图像对象的类型 | | `source`| 图像的来源 | | `retrieved_images`| 由CLIP检索得到的前5张候选图像序列 | 我们在此处发布了图像语料库，以供检索使用。 ## 联系方式 * 胡文博：[e-mail] ## 引用 @article{hu2024mragbench, title={MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models}, author={Hu, Wenbo and Gu, Jia-Chen and Dou, Zi-Yi and Fayyaz, Mohsen and Lu, Pan and Chang, Kai-Wei and Peng, Nanyun}, journal={arXiv preprint arXiv:2410.08182}, year={2024} }