BreastStage-Bench
收藏资源简介：
# BreastStage-Bench — A Workflow-Aligned Evaluation Benchmark for Breast Cancer MLLMs **BreastStage-Bench** is the held-out evaluation split of the **BreastStage** instruction corpus, released as a standalone dataset so that benchmarking can be done without pulling the (much larger) training corpus. It is the benchmark on which the companion model **BreastGPT** ([YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B)) is evaluated and reports 75.66% closed-end accuracy / 89.92% open-end score. The full training corpus lives in [**YYangYang/BreastStage**](https://www.modelscope.cn/datasets/YYangYang/BreastStage). ## Scope at a glance | | | | --- | --- | | **# files** | 25 JSON files | | **Modalities** | BUS, Mammography, CT, MRI, Histopathology (WSI) | | **Clinical stages** | screening / diagnosis / treatment | | **Task formats** | closed-ended VQA, open-ended VQA, ground caption | | **Patient-level holdout** | strict — no patient overlaps with the training corpus | --- ## Two evaluator-facing variants Every task is shipped in two media formats so that domain-specific medical models and general-purpose VLMs can be compared on equal footing: | Suffix | Target evaluator | Media format | | --- | --- | --- | | `*_NII.json` | **3D medical specialist models** (consume NIfTI volumes directly) | CT / MRI sequences as `.nii.gz` | | `*_H5.json` | **Pathology specialist models** (consume pre-extracted patch features) | WSI → `.h5` (CONCHv1.5 patch features) | | no suffix (`*.json`) | **General-purpose VLMs** (Qwen-VL / GPT-4V / default ms-swift inference, etc.) | CT / MRI → `.mp4` (slices stacked into video); WSI → 32 sampled `.png` patches; BUS / Mammography → single `.png` | BUS and Mammography are inherently 2D, so only the no-suffix variant exists for them. Within each `(modality, task)` group the two variants share their underlying samples — you can fairly compare any pair of models that consume one variant against the other. --- ## File listing All files follow the pattern `sampled_<Modality>-SWIFT_RESIZED_<TaskFamily>_Test[_FULL][_NII|_H5].json`. | Modality | Files | | --- | --- | | BUS | `Stage_GroundCaption`, `Stage_Lesion_Closed_VQA`, `Stage_Lesion_Open_VQA` (3 files) | | Mammography | `Stage_Mammo_Closed_VQA`, `Stage_Mammo_GroundCaption` (2 files) | | CT | `Screen_Closed_VQA`, `Screen_GroundCaption`, `Screen_Open_VQA` × {`.json`, `_NII.json`} (6 files) | | Histopathology | `Slide_Caption`, `Slide_VQA_Closed`, `Slide_VQA_Open` × {`.json`, `_H5.json`} (6 files) | | MRI | `Stage_JJ_Closed_VQA`, `Stage_JJ_GroundCaption`, `Stage_JJ_Open_VQA`, `Stage_JJ_Report` × {`_FULL.json`, `_FULL_NII.json`} (8 files) | MRI files keep only the `_FULL` variants (all sequences: T1 / T1dyn / DWI / ADC / …). --- ## JSON schema (ms-swift format) ```json { "messages": [ {"role": "system", "content": "You are an expert ..."}, {"role": "user", "content": "<image>\nDescribe ... Answer:"}, {"role": "assistant", "content": "(A) Hypoechoic"} ], "images": ["/nas/.../RESIZED/BUS/.../000015@0.png"], "topic": "EchoCharacteristics", "stage": "Screen" } ``` - For 3D modalities (CT / MRI) and multi-patch WSI cases, the field is `videos` rather than `images`. - The `system` content is the **stage-aware role prompt** used as a lightweight task router for BreastGPT. --- ## Path substitution > ⚠️ **This repository only contains evaluation annotations (JSON); raw images are not included.** Images must be obtained from the upstream sources (see [YYangYang/BreastStage](https://www.modelscope.cn/datasets/YYangYang/BreastStage) for the full source list and MRI application procedure). The `images` / `videos` fields are absolute paths on our local filesystem, laid out as `/nas/yangye.ly/breastGPT/datasets/RESIZED/<modality>/...` (the bench uses the **RESIZED** variant for radiology and pathology). Replace the prefix with your local image root: ```python import json, glob YOUR_IMAGE_ROOT = "/path/to/your/breast_images" # where you store the images locally for fp in glob.glob("BreastStage-Bench/*.json"): data = json.load(open(fp)) for ex in data: for k in ("images", "videos"): if k in ex and isinstance(ex[k], list): ex[k] = [p.replace("/nas/yangye.ly/breastGPT/datasets", YOUR_IMAGE_ROOT) for p in ex[k]] json.dump(data, open(fp, "w"), ensure_ascii=False) ``` --- ## Companion resources - **Training corpus** — [YYangYang/BreastStage](https://www.modelscope.cn/datasets/YYangYang/BreastStage) - **Companion model** — [YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B) - **Code (anonymous, under review)** — https://anonymous.4open.science/r/BreastGPT --- ## License & Citation - Released under **CC BY-NC 4.0** (non-commercial use). - When citing upstream image datasets, please follow each dataset's original citation requirements. ```bibtex @misc{breastgpt2026_anon, title = {BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine}, author = {Anonymous}, year = {2026}, note = {Under review. Code and data: \url{https://anonymous.4open.science/r/BreastGPT}}, howpublished = {\url{https://www.modelscope.cn/models/YYangYang/BreastGPT-8B}} } ``` --- ## Contact - Companion model: [YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B) - Training corpus: [YYangYang/BreastStage](https://www.modelscope.cn/datasets/YYangYang/BreastStage) - Data issues / feedback: please open an issue on the ModelScope repository. - **MRI image application**: please open an issue on the ModelScope repository or email the authors, stating your affiliation, intended research use, and whether the use is non-commercial; data will be provided after review and a signed DUA.
