RadImageNet-VQA

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/62cdea59a9be5c195561c2b8/JaS4YslW9wFR8dZ7LMawz.png" width="40%" alt="Raidium" /> </div> <hr> <h1 align="center">RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering</h1> <p align="center"> <a href="https://arxiv.org/abs/2512.17396"> <img src="https://cdn-uploads.huggingface.co/production/uploads/680266d8ef7a20a062c7e40d/eHFyXBbOf0pPfltvvMh5y.png" alt="arxiv-logo" width="50"/> </a> </p> <hr> We introduce RadImageNet-VQA, a large-scale dataset designed for training and benchmarking radiologic VQA on CT and MRI exams. Built from the CT/MRI subset of [RadImageNet](https://pubs.rsna.org/doi/full/10.1148/ryai.210315) and its expert-curated anatomical and pathological annotations, RadImageNet-VQA provides 750K images with 7.5M generated samples, including 750K medical captions for visual-text alignment and 6.75M question-answer pairs that span three radiology tasks: fine-grained pathology identification, anatomy recognition, and abnormality detection. The dataset includes open-ended, closed-ended, and multiple-choice questions across 8 anatomical regions and 97 pathologies, generated with prompt-based templates and constructed to probe visual-grounded understanding while minimizing text-only shortcut answering. For evaluation, we construct a stratified benchmark of 1,000 images with 9,000 question-answer pairs covering all tasks and question types. <div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/680266d8ef7a20a062c7e40d/bfgLBUgoUCWOJL4Ss-4qu.png" width="100%" alt="Raidium" /> </div> --- ## Data Creation RadImageNet-VQA was created to challenge multimodal models with tasks that demand radiology text-image understanding, pushing the boundaries of what these models can achieve in terms of perception and reasoning. The data for the RadImageNet-VQA dataset was build upon RadImageNet, a large expert-annotated medical imaging dataset in which each image is associated with a modality (CT, MRI, US), a body part (e.g., abdomen, hip, brain) and a pathology label. From this resource, we use the CT and MRI subsets to form the basis for generating clinically meaningful captions and VQA samples across anatomy, abnormality, and fine-grained pathology tasks. <div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/680266d8ef7a20a062c7e40d/GZGe4kQ5PeZwk8p4Ola_D.png" width="100%" alt="Raidium" /> </div> ## Zero-shot Evaluation on RadImageNet-VQA **Zero-shot accuracies (%) of VLMs on RadImageNet-VQA benchmark.** Results are reported across anatomy recognition, abnormality detection (*Abn*), and pathology identification using four question formats: *Open* (free-form), *Closed+* (always 'yes' as true answer), *Closed–* (always 'no'), and *MC* (multiple-choice).  **Bold** = best, *italic* = second best ## Data Structure ### Alignment Data The alignment component contains single caption samples per image, intended to align visual content with concise clinical descriptions. Each instance conceptually includes: - an image - a single prompt–response pair - structured metadata **Fields:** - `id`: unique sample identifier - `image`: relative path to the medical image - `conversations`: one human prompt and one descriptive response - `metadata`: modality, anatomical location, abnormality flag, pathology label The response provides a brief clinical description of the image. --- ### Instruction Data The instruction component contains multiple question–answer pairs per image and is intended for instruction tuning of multimodal models. Each instance includes: - an image - one or more QA-style conversation turns - structured metadata describing the task Supported instruction types include image description, pathology identification, modality recognition, and anatomical localization. --- ### Benchmark Data The benchmark split is designed for standardized evaluation of medical VQA models. It contains 9,000 question–answer pairs across 1,000 images and includes three question types: - open-ended (free-form answers) - closed-ended (yes/no) - multiple-choice (options A–D) **Benchmark fields:** - `image`: medical image reference - `question`: question presented to the model - `choices`: answer options (multiple-choice only) - `answer`: ground-truth answer - `question_type`: open, yes/no, or multiple-choice - `metadata`: modality, anatomy, pathology, and correctness labels --- ### Metadata Metadata fields provide structured clinical and contextual information: - `modality`: imaging modality (e.g., CT, MRI) - `location`: anatomical region - `is_abnormal`: presence of pathology - `pathology`: pathology category - `content_type`: task type (description, pathology, etc.) - `question_id`: question template identifier - `correct_text`: textual form of the correct answer (when applicable) ### Data Splits The dataset is organized into three configurations with training and validation splits: | | Alignment | | Instruction Tuning | | Benchmark | |-------------------------|:---------:|:---------:|:------------------:|:---------:|:---------:| | | Train | Validation | Train | Validation | Test | | Samples | 750,009 | 83,668 | 750,009 | 83,668 | 9,000 | | Images | 750,009 | 83,668 | 750,009 | 83,668 | 1,000 | | QAs per image | 1 | 1 | ~9 | ~9 | 9 | | Total QAs | 750K | 83K | 6.75M | 753K | 9K | ## Acknowledgments The dataset is built upon RadImageNet https://www.radimagenet.com/. ## Citation ``` @article{butsanets2025radimagenet, title={RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering}, author={Butsanets, L{\'e}o and Corbi{\`e}re, Charles and Khlaut, Julien and Manceron, Pierre and Dancette, Corentin}, journal={arXiv preprint arXiv:2512.17396}, year={2025} } ```