ClinFusion-Eval-Data

<div align="center"> # 🏥 ClinFusion-Eval-Data **The Holistic Evaluation Suite for Vision-Centric Medical Multimodal LLMs** [](https://arxiv.org/abs/2607.24743) [](https://github.com/Alibaba-DAMO-Academy/ClinFusion) [](LICENSE) </div> **ClinFusion-Eval-Data** is the unified evaluation corpus used to benchmark the [ClinFusion](https://github.com/Alibaba-DAMO-Academy/ClinFusion) model series ([ClinFusion-8B](https://huggingface.co/Alibaba-DAMO-Academy/ClinFusion-8B), [ClinFusion-32B](https://huggingface.co/Alibaba-DAMO-Academy/ClinFusion-32B)). It packages **211,810 evaluation records** spanning 22 public medical benchmarks into a single, consistently-formatted suite, together with **509 GiB of the underlying 2D images and native 3D CT volumes** they refer to. The goal is reproducibility: instead of re-downloading and re-normalising two dozen heterogeneous benchmarks, you get one media archive, one path convention, and one record schema covering multiple-choice, open-ended, and report-generation evaluation. --- ## 🌟 What's Inside * **Three evaluation tracks**: general 2D multimodal & text-only medical QA, native 3D volumetric CT reasoning, and an instruction-following stress set. * **Unified record schema**: every record carries a `source`, an `eval_method`, and media paths that resolve directly into the extracted `mm_data/` tree. * **Rich difficulty metadata**: the general track is tagged with clinical `task` type, `organs_involved`, `sample_quality`, and `hardness`, so results can be sliced by capability instead of reported as one aggregate number. * **Native 3D support**: 3D questions reference NIfTI (`.nii.gz`) volumes, including multi-phase longitudinal series for temporal diagnosis. --- ## 📁 Files | File | Records | Description | |---|---|---| | `general_eval_data_with_tags_v1.jsonl` | 158,661 | 2D multimodal + text-only medical QA across 19 sources, fully tagged | | `3d_eval_data_ct-rate_amos_3d-rad.jsonl` | 52,346 | Native 3D CT reasoning over CT-RATE, AMOS-MM and 3D-RAD | | `3d_eval_data_ct-rate_amos_3d-rad_lite.jsonl` | 4,000 | Balanced 4k subset of the 3D track for fast iteration | | `if_eval_data_no_staging.jsonl` | 803 | Instruction-following set spanning 7 answer formats | | `mm_data.tar.part-000` … `part-101` | — | The media archive: 509.25 GiB of images and CT volumes, split into 102 shards | The `_lite` file is a subset of the full 3D track, so record counts are not additive. The four jsonl files can be accessed in [ClinFusion Github](https://github.com/alibaba-damo-academy/ClinFusion); this repository hosts the media archive they point to. ### Why the media is sharded The Hub caps single files at 50 GB, so `mm_data.tar` (509.25 GiB) ships as 102 byte-exact shards of 5 GiB (the last one is 4.25 GiB). Concatenating them reproduces the original archive bit-for-bit. --- ## 🚀 Quick Start ### 1. Download and reassemble the media ```bash huggingface-cli download Alibaba-DAMO-Academy/ClinFusion-Eval-Data \ --repo-type dataset --local-dir ClinFusion-Eval-Data cd ClinFusion-Eval-Data cat mm_data.tar.part-* | tar xf - # -> ./mm_data/ ``` Reassembly needs ~1 TB of free space while both the shards and the extracted tree are present; delete the shards afterwards to halve that. Verify the concatenated archive before extracting if you prefer: ```bash cat mm_data.tar.part-* | wc -c # expect 546797690880 ``` ### 2. Load the annotations The jsonl files ship with the [ClinFusion GitHub repository](https://github.com/Alibaba-DAMO-Academy/ClinFusion); clone it alongside the extracted media. ```python import json records = [json.loads(line) for line in open("general_eval_data_with_tags_v1.jsonl")] sample = records[0] print(sample["source"]) # chexpert_plus print(sample["eval_method"]) # report_generation print(sample["images_path"]) # ['mm_data/CheXpert_Plus/valid/patient64620/study1/view1_frontal.png'] ``` Media paths are relative to the directory holding `mm_data/`, so run evaluation from the extraction root and they resolve as-is. For end-to-end inference and scoring scripts, see the **[ClinFusion GitHub repository](https://github.com/Alibaba-DAMO-Academy/ClinFusion)**. --- ## 🧬 Record Schema **2D / text track** (`general_eval_data_with_tags_v1.jsonl`) | Field | Description | |---|---| | `source` | Originating benchmark, e.g. `omnimedvqa`, `medxpertqa_mm`, `chexpert_plus` | | `index` | Index within that source | | `eval_method` | `mcq`, `mcq_context`, `open`, or `report_generation` | | `language` | `english` (154,202) or `chinese` (4,459) | | `ground_truth` | Reference answer, option letter, or full reference report | | `images_path` | List of image paths under `mm_data/`; empty for text-only items | | `meta_info` | Source-specific context (view, projection, patient demographics, …) | | `organs_involved` | Comma-separated organ tags, `N/A` for text-only items | | `task` | Clinical capability being probed (see distribution below) | | `sample_quality` | `good` / `mild` / `bad`, from an automated quality audit | | `hardness` | `easy` / `mild` / `hard` | **3D track** (`3d_eval_data_*.jsonl`) replaces `images_path` with `ct_path` (NIfTI volumes), carries `question` plus optional `options`, and uses `meta_info.task` for the 3D capability label. **Instruction-following track** (`if_eval_data_no_staging.jsonl`) carries `question`, `options`, and `images_path`, with `eval_method` selecting the required answer format. --- ## 📊 Composition **2D / text sources** (19) | Source | Records | Source | Records | |---|---|---|---| | `omnimedvqa` | 88,996 | `medxpertqa_mm` | 2,000 | | `pmcvqa` | 33,430 | `mmlu_med` | 1,871 | | `path_vqa` | 6,719 | `medqa_usmle` | 1,273 | | `gmaimmbench` | 4,550 | `pubmedqa` | 500 | | `medmcqa` | 4,183 | `vqa_rad` | 451 | | `medqa_mcmle` | 3,426 | `medbullets_op4` | 308 | | `medframeqa` | 2,851 | `medbullets_op5` | 308 | | `supergpqa_med` | 2,755 | `iuxray` | 296 | | `medxpertqa_text` | 2,450 | `chexpert_plus` | 200 | | `slake` | 2,094 | | | **3D sources**: `3d_rad` (33,910), `ct_rate_vqa` (15,249), `amos-mm` (3,187). **Clinical task distribution** (2D / text track) | Task | Records | Task | Records | |---|---|---|---| | `lesion_understanding` | 76,061 | `clinical_knowledge` | 4,242 | | `organ_understanding` | 25,905 | `examination_planning` | 2,041 | | `modality_recognition` | 23,477 | `drug_usage` | 1,520 | | `basic_science_knowledge` | 15,727 | `disease_staging` | 468 | | `etiological_diagnosis` | 7,994 | `protocol_design` | 442 | | | | `change_comparison` / `treatment_response` | 306 / 282 | **3D capability distribution**: `Existence_Detection` (23,472), `Static_Temporal_Diagnosis` (2,873), `Longitudinal_Temporal_Diagnosis` (2,873), `Anomaly_Detection` (2,666), `Image_Observation` (1,024), `Medical_Computation` (1,002). **Evaluation formats**: `mcq` dominates the 2D track (152,850), with `open` (4,815), `mcq_context` (500) and `report_generation` (496). The instruction-following track spreads 803 items across `mcq`, `mcq_context`, `open`, `report_generation`, `prognosis`, `treatment` and `seer`. **Media formats**: media references in the general track resolve to PNG (90,081), JPG/JPEG (56,051), TIF (2,573) and BMP (431) files; all 3D items reference NIfTI `.nii.gz` volumes. **Difficulty and quality**: `hardness` splits into mild (100,468), easy (57,547) and hard (646); `sample_quality` flags 1,171 items as `bad` and 24 as `mild`, which you may exclude for a cleaner comparison. --- ## ⚖️ Licensing and Intended Use This release is distributed under Apache-2.0, but **each constituent benchmark remains under its own original license and terms of use**. Before redistributing or using any subset commercially, consult the upstream license for the relevant `source` (CT-RATE, AMOS-MM, OmniMedVQA, PMC-VQA, CheXpert Plus, MedXpertQA, and the others listed above). The data is provided **for research evaluation only**. It is not a medical device, contains de-identified third-party clinical data, and must not be used for diagnosis or treatment decisions. --- ## 📄 Citation If you find ClinFusion useful in your research, please consider citing: ```bibtex @article{yuan2026ClinFusion, title={ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding}, author={Yuan, Hangjie and Qian, Yichen and Tang, Zhiwei and Xu, Xianzhe and Wu, Lirong and Yang, Sicheng and Wang, Jinwang and Wang, Pengju and Zeng, Zhitao and Han, Yizeng and Xing, Yan and Luo, Shengxuan and Feng, Tao and Xie, Qing and Yao, Weigen and Yang, Yi and Liu, Zuozhu and Tang, Jiasheng and Wang, Shaocheng and Wang, Jitao and Dong, Jiahong and Chen, Weihua and Xu, Feng and Wang, Fan}, journal={arXiv preprint arXiv:2607.24743}, year={2026} } ``` Please also cite the individual benchmarks you report results on. --- ## 📞 Contact & Acknowledgements Built with ❤️ by **Alibaba DAMO Academy**. This suite would not exist without the teams behind the public benchmarks it aggregates — our thanks to all of them for releasing their data to the community.