BreastStage

# BreastStage — A Workflow-Aligned Instruction Corpus for Breast Cancer **BreastStage** is a multimodal instruction-tuning corpus aligned with the **full breast-cancer clinical workflow (screening → diagnosis → treatment)**, released alongside the unified multimodal LLM **BreastGPT** ([YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B)). The corpus is organized around the cognitive demands of real clinical workflows, covering the full task spectrum from breast screening and diagnostic imaging to histopathology-based treatment planning. ## Scope at a glance | | | | --- | --- | | **Instruction-following pairs** | ≈ **1.86 M** | | **Unique 2D / 3D images** | ≈ 662 K | | **Records with bbox / mask** | ≈ 606 K | | **Imaging modalities** | **5** — BUS, Mammography, CT, MRI, Histopathology (WSI) | | **Sub-datasets** | **17** (public + private MRI from collaborating hospitals) | | **Task templates** | **136** (expert-designed) | | **Task formats** | classification, visual grounding, closed-ended VQA, open-ended VQA, ground caption, structured report generation | | **Held-out evaluation split** | released as a separate dataset: [**YYangYang/BreastStage-Bench**](https://www.modelscope.cn/datasets/YYangYang/BreastStage-Bench) | Samples are distributed across clinical stages as **Screening 57.9% / Diagnosis 36.7% / Treatment 5.4%**, broadly tracking the relative data availability of each stage in real-world breast care. ### Stage-aligned task taxonomy | Stage | # Pairs | Task families (selected) | | --- | --- | --- | | **Screening** | 1,075,092 | modality / view / laterality recognition; breast-tissue characterization (FGT / BPE / density); risk and decision tasks; lesion presence / morphology; post-surgical change tracking | | **Diagnosis** | 680,409 | BI-RADS and pathology classification; lesion-level signal & kinetic characterization; associated findings (lymph-node involvement, tissue invasion); tumour characterization and staging | | **Treatment** | 100,567 | prognosis prediction; surgical planning; systemic therapy and biomarker assessment; WSI-based pathology subtyping | --- > ⚠️ **This repository only contains instruction annotations (JSON); raw images are not included.** Due to data-distribution agreements, users must obtain images from the original upstream sources. The `image_path` / `mask_path` / `path` / `images` / `videos` fields in the JSON are absolute paths on our local filesystem — please follow the [Path Substitution](#path-substitution) section below to remap them. > > 📩 **About MRI images**: the MRI subset comes from **two collaborating clinical institutions** as multi-parametric MRI data, with T1 / T1dyn lesion masks manually annotated by 10 breast specialists. The instruction annotations are released here, but **the corresponding images cannot be downloaded directly — they require a separate application and a signed Data Use Agreement (DUA)** before they are provided by the authors. See [Contact](#contact) for the application procedure. --- ## Directory Layout ``` BreastStage/ ├── BUS/ # Ultrasound — 16 files ├── Mammography/ # Mammography — 8 files ├── CT/ # CT — 12 files ├── Histopathology/ # WSI pathology — 12 files └── MRI/ # MRI — 16 files (images require application; see above) ``` The held-out evaluation split is released as a separate dataset, [**YYangYang/BreastStage-Bench**](https://www.modelscope.cn/datasets/YYangYang/BreastStage-Bench), so that benchmarking workflows can pull it independently of the (much larger) training corpus. **Strict patient-level separation** is enforced — no patient appears on both sides — so downstream evaluation reflects clinical generalization rather than memorization of shared visual content. ## File-naming conventions Within each modality directory, **the same underlying data is provided in several variants** so you can pick the format you need: | Prefix / Suffix | Meaning | | --- | --- | | `Stage_*_Train.json` / `Stage_*_Test.json` | **Raw annotations** with rich metadata (Question topic / Dataset / bbox / System prompt, etc.); paths point to original images | | `*_DownSample.json` | Class-balanced down-sampled variant — mitigates long-tail distribution | | `SWIFT_RESIZED_*.json` | Converted to the [ms-swift](https://github.com/modelscope/ms-swift) training format (`messages` + `images` / `videos`); media paths point to the resized version | Task-family tokens (`Lesion` / `Mammo` / `Screen` / `Slide` / `Stage_JJ`) identify different task templates, e.g.: - BUS: `Stage_Lesion_Closed_VQA_*` (lesion closed-end), `Stage_GroundCaption_*` (grounding caption), `Stage_Report_VQA_*` (report generation) - Mammography: `Stage_Mammo_Closed_VQA_*`, `Stage_Mammo_GroundCaption_*` - CT: `Screen_Closed_VQA_*`, `Screen_Open_VQA_*`, `Screen_GroundCaption_*` - Histopathology: `Slide_VQA_Closed_*`, `Slide_VQA_Open_*`, `Slide_Caption_*` - MRI: `Stage_JJ_Closed_VQA_*`, `Stage_JJ_Open_VQA_*`, `Stage_JJ_GroundCaption_*`, `Stage_JJ_Report_*`; the `_FULL` suffix indicates that the file contains the full set of sequences (T1 / T1dyn / DWI / ADC, etc.) --- ## JSON schema ### Raw format (`Stage_*` / `Screen_*` / `Slide_*`) ```json { "stage": "Screen", // screening / diagnosis / treatment "VQA type": "closed-end", // closed-end / open-end / caption / report "iid": 15, "fid": "000015@0", // case / lesion ID "image_path": "/nas/.../Images/BUS/.../[e-mail]", "mask_path": "/nas/.../Images/BUS/.../[e-mail]", // optional "bbox": [177, 123, 258, 184], // optional, xyxy "Dataset": "US_Lesion", // upstream source "Question topic": "EchoCharacteristics", "Question type": "single choice", "Question": "Describe the internal echo texture of the mass.", "options": ["Anechoic", "Complex", "Hyperechoic", "Hypoechoic", "Isoechoic"], "Answer": "Hypoechoic", "qid": 343, "System": "You are an expert screening breast radiologist ..." } ``` > Fields vary slightly by modality (e.g. Mammography adds `split` / `laterality`, Histopathology uses `pid` / `task`). When in doubt, `json.load` a file and inspect `keys()`. ### ms-swift format (`SWIFT_*`) ```json { "messages": [ {"role": "system", "content": "You are an expert ..."}, {"role": "user", "content": "<image>\nDescribe ... Answer:"}, {"role": "assistant", "content": "(A) Hypoechoic"} ], "images": ["/nas/.../RESIZED/BUS/.../[e-mail]"], "topic": "EchoCharacteristics", "stage": "Screen" } ``` Can be fed directly as the [ms-swift](https://github.com/modelscope/ms-swift) `--dataset` input. For 3D modalities (CT / MRI) and multi-patch WSI cases, the SWIFT files use `videos` instead of `images`. --- ## Path substitution All `image_path` / `mask_path` / `images` / `videos` fields in the JSON are absolute paths on our local filesystem, laid out as: ``` /nas/yangye.ly/breastGPT/datasets/Images/<modality>/<source>/... # original images /nas/yangye.ly/breastGPT/datasets/RESIZED/<modality>/<source>/... # resized version (used by SWIFT_RESIZED_* / Bench) ``` Replace the prefix in bulk with your local image root: ```python import json, glob YOUR_IMAGE_ROOT = "/path/to/your/breast_images" # where you store the images locally for fp in glob.glob("BreastStage/**/*.json", recursive=True): data = json.load(open(fp)) for ex in data: for k in ("image_path", "mask_path", "path"): if k in ex and isinstance(ex[k], str): ex[k] = ex[k].replace("/nas/yangye.ly/breastGPT/datasets", YOUR_IMAGE_ROOT) for k in ("images", "videos"): if k in ex and isinstance(ex[k], list): ex[k] = [p.replace("/nas/yangye.ly/breastGPT/datasets", YOUR_IMAGE_ROOT) for p in ex[k]] json.dump(data, open(fp, "w"), ensure_ascii=False) ``` --- ## Data sources | Modality | Primary sources | Annotation source | | --- | --- | --- | | BUS | Public BUS datasets (curated as BUS-CoT) | Expert hand-drawn lesion masks | | Mammography | BMCD, CBIS-DDSM, CMMD, CSAW-M, EMBED | Bounding boxes released by EMBED | | CT | CT-RATE and other public CT collections | Automatic tumour segmentation via DRT-M3D | | Histopathology | TCGA-BRCA and other public WSI datasets | Slide-level labels; CONCHv1.5 patch features | | MRI | **2 collaborating clinical institutions** | Manual T1 / T1dyn annotations by 10 breast specialists | Please follow the respective license / DUA of each upstream public dataset. MRI images are provided by the authors upon application approval. --- ## Companion model: BreastGPT Short summary (see paper §Method for full details): - **Backbone** — Qwen3-VL; the LLM trunk is preserved and stage-aware system prompts act as lightweight task routers. - **Modality-aware resolution gating** — WSI inputs are automatically routed to the GigaPixel branch; all other radiological modalities flow through the Standard branch. - **GigaPixel branch** — Frozen CONCHv1.5 patch encoder + trainable LongNet aggregator (dilated attention, avoiding the $O(N^2)$ bottleneck). - **Universal concept-based token selector** — A coverage-maximization objective adapted from MMTok compresses multi-scale visual evidence into a fixed token budget. - **On BreastStage-Bench** — 75.66% closed-end accuracy / 89.92% open-end score, outperforming both general-purpose and medical-specific MLLMs. Model weights: [YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B) --- ## License & Citation - The instruction annotations are released under **CC BY-NC 4.0** (non-commercial use). - When citing upstream image datasets, please follow each dataset's original citation requirements. If you use this dataset or the BreastGPT model, please cite the following (the paper is currently under review; the final citation will be updated after acceptance): ```bibtex @misc{liu2026breastgptmultimodallargelanguage, title={BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine}, author={Yang Liu and Jiajin Zhang and Danyang Tu and Yaojun Hu and Jiao Qu and Jiuyu Zhang and Yu Shi and Wei Fang and Shi Gu and Ling Zhang and Yingda Xia}, year={2026}, eprint={2606.04911}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2606.04911}, } ``` --- ## Contact - Companion model: [YYangYang/BreastGPT-8B](https://www.modelscope.cn/models/YYangYang/BreastGPT-8B) - Data issues / feedback: please open an issue on the ModelScope repository. - **MRI image application**: please open an issue on the ModelScope repository or email the authors, stating your affiliation, intended research use, and whether the use is non-commercial; data will be provided after review and a signed DUA.