foreverbeliever/OmniMedVQA

---
viewer: false
---

# [OmniMedVQA](https://arxiv.org/abs/2402.09181)

We introduce **OmniMedVQA**, large-scale and comprehensive Visual Question Answering benchmark tailored
to the medical domain. This benchmark is collected from **73 different medical datasets**, contains **118,010 images** with **127,995 QA-items**, covering **12 different medical image modalities** and referring to more than **20 human anatomical regions**. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs.

Please visit the [GitHub]() page and further exploit the dataset!

OmniMedVQA is built using multiple publicly available datasets. Due to data privacy and ethical requirements, **we currently only offer access to a subset of OmniMedVQA**. For open-access datasets, we provide the image and the corresponding QA information constructed on these datasets. For restricted-access datasets, we provide the relative paths to the image data in the original dataset, along with the QA information we have constructed. Please adhere to the respective dataset's license to access and use this data. We will continue to update and maintain this database. Please stay tuned for further updates from us.

## 👉 Filesystem Hierarchy

```bash
~/OmniMedVQA
├── Images
| ├── ACRIMA
| | ├── Im002_ACRIMA.png
| | └── ...
| ├── Adam Challenge
| └── ...
├── QA_information
| ├── Open-access
| | ├── ACRIMA.json
| | ├── Adam Challenge.json
| | └── ....
| └── Restricted-access
| ├── AIDA.json
| ├── AIROGS.json
| └── ...
|
└──README.md

```

In the **Images** directory, we only provide the images from the open-access dataset. In the **QA_information** directory, we provide QA information for both open-access and restricted-access data in the form of JSON files. The QA information includes image paths, corresponding image modality types, questions, question types, options, ground truth answers, and corresponding dataset name. The content is highly comprehensive. Here is an example of the information provided:

```
{
"dataset": "Covid CT",
"question_id": "Covid CT_0082",
"question_type": "Anatomy Identification",
"question": "What anatomical area is shown in this picture?",
"gt_answer": "Chest region.",
"image_path": "Images/Covid CT/CT_COVID/bmj.m606.full-p4-22%3.png",
"option_A": "Upper arm region",
"option_B": "Chest region.",
"option_C": "Leg region",
"option_D": "Shoulder and upper back region",
"modality_type": "CT(Computed Tomography)"
},
```

The OmniMedVQA dataset contains a portion of medical images that have been split from 3D data. The naming convention for these sliced images follows the following format:

```bash
{dataset name}/{ori name}_{dimension_slice}.png
```

Note: "dataset name" refers to the specific dataset name that the case is from. "ori name" is the original case name in its dataset. "dimension slice", e.g., "x_100", indicates the dimension along which we split a 3D case as well as the slice ID in this dimension. If we split a 3D case with axis x and the current slice is 100, then the term can be "x_0100".

## 🤝 免责声明

- OmniMedVQA是基于多个公开的数据集构建，旨在取之于社区，回馈于社区，为研究人员和开发者提供一个用于学术和技术研究的资源。使用本数据集的任何个人或组织（以下统称为“使用者”）需遵守以下免责声明：

1. 数据集来源：本数据集基于多个公开的数据集构建，这些数据集的来源已在论文中明确标明。使用者应当遵守原始数据集的相关许可和使用条款。
2. 数据准确性：尽管我们已经努力确保数据集的准确性和完整性，但使用者应自行承担使用数据集可能带来的风险和责任。
3. 责任限制：在任何情况下，数据集的提供者及相关贡献者均不对使用者的任何行为或结果承担责任。
4. 使用约束：使用者在使用本数据集时，应遵守适用的法律法规和伦理规范。使用者不得将本数据集用于非法、侵犯隐私、诽谤、歧视或其他违法或不道德的目的。
5. 知识产权：本数据集所有影像数据的知识产权归原始数据集的相关权利人所有，使用者不得以任何方式侵犯数据集的知识产权。

- 作为非盈利机构，团队倡导和谐友好的开源交流环境，若在开源数据集内发现有侵犯您合法权益的内容，请联系我们，我们将尽最大努力协助您处理。

- 通过下载、复制、访问或使用本数据集，即表示使用者已阅读、理解并同意遵守本免责声明中的所有条款和条件。如果使用者无法接受本免责声明的任何部分，请勿使用本数据集。

## 🤝 Disclaimer

- OmniMedVQA is constructed based on multiple publicly available datasets and aims to provide a resource for academic and technical research to researchers and developers. Any individual or organization (hereinafter referred to as "User") using this dataset must comply with the following disclaimer:

1. Dataset Source: OmniMedVQA is constructed based on multiple publicly available datasets, and the sources of these datasets have been clearly indicated in the paper. Users should adhere to the relevant licenses and terms of use of the original datasets.
2. Data Accuracy: While efforts have been made to ensure the accuracy and completeness of the dataset, users assume all risks and liabilities associated with the use of the dataset.
3. Limitation of Liability: Under no circumstances shall the dataset providers or contributors be held liable for any actions or outcomes of the Users.
4. Usage Constraints: Users must comply with applicable laws, regulations, and ethical norms when using this dataset. The dataset must not be used for illegal, privacy-infringing, defamatory, discriminatory, or other unlawful or unethical purposes.
5. Intellectual Property: The intellectual property rights of the image data in this dataset belong to the relevant rights holders of the original datasets. Users must not infringe upon the intellectual property rights of the dataset in any way.

- As a non-profit organization, we advocate for a harmonious and friendly open-source communication environment. If any content in the open dataset is found to infringe upon your legitimate rights and interests, please contact us and we will make our best effort to assist you in addressing the issue.

- By downloading, copying, accessing, or using this dataset, the User indicates that they have read, understood, and agreed to comply with all the terms and conditions of this disclaimer. If the User cannot accept any part of this disclaimer, please refrain from using this dataset.

## 🤝 Acknowledgement

- We thank all medical workers and dataset owners for making public datasets available to the community. If you find that your dataset is included in our OmniMedVQA but you do not want us to do so, please contact us to remove it.

## Reference

```
@article{hu2024omnimedvqa,
title={OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM},
author={Hu, Yutao and Li, Tianbin and Lu, Quanfeng and Shao, Wenqi and He, Junjun and Qiao, Yu and Luo, Ping},
journal={arXiv preprint arXiv:2402.09181},
year={2024}
}
```

搜集汇总

数据集介绍

构建方式

在医学视觉问答领域，OmniMedVQA数据集的构建体现了对多源异构数据的系统性整合。该数据集从73个不同的医学数据集中精心遴选，共汇集了118,010张医学图像与127,995个问答对，覆盖了包括计算机断层扫描、磁共振成像在内的12种医学影像模态，并涉及超过20个人体解剖区域。构建过程中，研究团队严格遵循数据隐私与伦理规范，对开放获取数据集直接提供图像及对应问答信息，而对受限访问数据集则仅提供原始数据路径与构建的问答标注，确保学术使用的合规性。

特点

OmniMedVQA的显著特征在于其规模宏大且内容全面，所有图像均源自真实临床场景，确保了与医学领域需求的高度契合。数据集不仅囊括了多样化的影像模态与解剖结构，还提供了丰富的问答信息，包括问题类型、选项、标准答案及影像模态描述，形成了结构化的JSON格式标注。特别值得注意的是，对于从三维数据中分割的二维切片图像，数据集采用了规范的命名体系，便于研究者追溯原始数据与切片维度，这为大规模语言视觉模型在医学领域的评估提供了坚实且细致的基准。

使用方法

使用OmniMedVQA数据集时，研究者需首先遵循其文件系统层次结构，在Images目录下获取开放访问的图像，并在QA_information目录下查阅对应的JSON格式标注文件。每个标注条目均包含图像路径、问题、答案及元数据，支持直接用于模型训练与评估。对于涉及受限访问数据的研究，用户需根据标注中的路径指引，自行依据原始数据集的许可协议获取相应图像。该数据集的设计旨在服务于学术研究，使用者应严格遵守免责声明中的各项条款，确保在符合伦理与法律规范的框架内开展探索。

背景与挑战

背景概述

在医学人工智能领域，视觉问答任务旨在评估模型对医学影像的理解与推理能力，是推动辅助诊断技术发展的关键环节。OmniMedVQA数据集由香港大学等机构的研究团队于2024年创建，其核心研究问题在于构建一个大规模、多模态的医学视觉问答基准，以全面评估大型视觉语言模型在真实医疗场景中的性能。该数据集整合了73个不同医学数据集，涵盖12种影像模态和超过20个人体解剖区域，以其丰富的多样性和临床真实性，为医学人工智能研究提供了重要的评估工具，显著提升了该领域模型测试的标准化与严谨性。

当前挑战

OmniMedVQA数据集面临的挑战主要体现在两个方面：在领域问题层面，医学视觉问答需应对跨模态影像的复杂语义理解，如从CT、MRI等多种成像技术中提取关键诊断信息，并回答涉及解剖识别、病理描述等专业问题，这对模型的跨域知识融合与细粒度推理提出了极高要求。在构建过程中，挑战源于数据整合的复杂性，包括协调不同数据源的许可协议与隐私规范，处理三维医学影像的切片标准化，以及确保问答对在多样医学语境下的准确性与一致性，这些因素均增加了数据集构建的技术与伦理难度。

常用场景

经典使用场景

在医学人工智能领域，视觉问答任务对模型的多模态理解能力提出了严峻挑战。OmniMedVQA作为大规模综合性基准，其经典使用场景在于评估大型视觉语言模型在真实医疗环境下的诊断推理能力。该数据集通过整合73个不同医学数据集、涵盖12种影像模态和20余个人体解剖区域，为研究者提供了标准化的测试平台，用以检验模型从复杂医学图像中提取关键信息并回答专业问题的准确性。

解决学术问题

医学视觉问答长期面临数据稀缺、模态单一和评估标准不统一等学术难题。OmniMedVQA通过构建覆盖广泛医学领域的大规模基准，有效解决了跨模态医学知识融合的评估困境。该数据集促进了模型在解剖结构识别、病理特征描述和临床决策推理等方面的研究，为探索通用医学人工智能系统提供了关键的数据支撑，推动了多模态医学分析向更全面、更可靠的方向发展。

衍生相关工作

OmniMedVQA的发布催生了一系列围绕医学多模态理解的创新研究。相关经典工作包括基于该基准的模型性能评估框架、针对医学领域适配的视觉语言模型微调方法，以及跨模态医学知识迁移技术。这些研究不仅深化了对模型在专业领域能力边界的认识，还促进了如医学报告自动生成、智能问诊系统等应用方向的探索，形成了以数据驱动为核心的医学人工智能研究新范式。