ClinSeek-Bench

Name: ClinSeek-Bench
Creator: maas
Published: 2026-05-21T10:42:21+08:00

魔搭社区2026-08-11 更新2026-08-16 收录

数据链接：

https://modelscope.cn/datasets/UCSC-VLAA/ClinSeek-Bench 数据链接链接失效反馈

官方服务：

资源简介：

# ClinSeek-Bench ClinSeek-Bench is the evaluation suite introduced in [ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning](https://arxiv.org/abs/2605.20176). It evaluates clinical reasoning under two paired settings with the same task definitions and answer labels: - **Curated Input**: the model answers from the evidence package provided by the source benchmark. - **Automated Evidence-Seeking**: the curated context is removed, and the model must retrieve evidence from raw clinical data using ClinSeekAgent tools. This Hugging Face dataset release provides the metadata needed to reconstruct ClinSeek-Bench. It is not a fully materialized benchmark package. To rebuild the full benchmark locally from this released metadata, see [Reconstructing The Full Benchmark](#reconstructing-the-full-benchmark). ## Why This Release Contains Metadata Only ClinSeek-Bench is built from credentialed clinical datasets, including MIMIC-IV, MIMIC-IV-Note, MIMIC-IV-ED, MIMIC-CXR, MIMIC-CXR-JPG, EHRXQA, and MedMod. These sources contain protected clinical information and must be obtained under each user's own credentialed access and data-use agreements. For privacy and licensing reasons, this repository does **not** redistribute: - raw MIMIC tables; - patient-level SQLite databases; - chest X-ray image files; - radiology report text; - experiment logs or model trajectories. Instead, this repository releases the metadata needed to rebuild the complete runtime benchmark locally from the official source datasets. ## Released Metadata | Split | Released metadata | Rows | Modality | What it contains | What it excludes | | --- | --- | ---: | --- | --- | --- | | Text-only | `ClinSeek-Bench_text.json` | 1,800 | EHR | `qid`, `subject_id`, `hadm_id`, prediction-time metadata, task names, questions, labels, candidates | patient SQLite DBs and raw MIMIC rows | | Multimodal | `ClinSeek-Bench_multimodal.jsonl` | 989 | EHR + CXR | released `qid`s, questions, normalized labels, patient/time metadata, CXR image references, report references | CXR JPG files, report text, rendered EHR contexts, patient SQLite DBs | The metadata files are the source of truth for reconstruction and evaluation. During local reconstruction, the ClinSeekAgent scripts materialize protected runtime assets such as patient databases, rendered EHR contexts, linked CXR files, and radiology reports from the official source datasets. ## Benchmark Composition ### Text-only EHR tasks The text-only split is derived from EHR-Bench in [EHR-R1](https://github.com/MAGIC-AI4Med/EHR-R1). It contains 45 EHR subtasks covering risk-prediction and decision-making scenarios. We sample 40 examples per subtask, resulting in 1,800 text-only examples. The released metadata contains 1,563 unique patients. ### Multimodal tasks The multimodal split adapts EHRXQA and MedMod, both grounded in MIMIC-IV EHRs and MIMIC-CXR chest radiographs. It contains 989 image-grounded examples: - 497 EHRXQA-derived CXR question-answering rows; - 492 MedMod-derived ICU/CXR prediction rows. The six multimodal task groups are CXR finding presence, CXR finding enumeration, CXR temporal change comparison, 24-hour decompensation prediction, in-hospital mortality prediction, and phenotype prediction. ## Experimental Results We evaluate ClinSeekAgent under the **Automated Evidence-Seeking** setting and compare it with the paired **Curated Input** setting. All numbers below are F1 scores in percentage points. `Δ` is Automated Evidence-Seeking minus Curated Input. ### Text-only EHR tasks <table> <thead> <tr> <th rowspan="2">Model</th> <th colspan="3">Risk Prediction</th> <th colspan="3">Decision Making</th> <th colspan="3">Overall</th> </tr> <tr> <th>ClinSeekAgent</th> <th>Curated Input</th> <th>Δ</th> <th>ClinSeekAgent</th> <th>Curated Input</th> <th>Δ</th> <th>ClinSeekAgent</th> <th>Curated Input</th> <th>Δ</th> </tr> </thead> <tbody> <tr><td colspan="10">Closed-source models</td></tr> <tr> <td>Claude Opus 4.6</td> <td align="right">90.7</td><td align="right">81.0</td><td align="right">+9.7</td> <td align="right">44.8</td><td align="right">45.9</td><td align="right">-1.1</td> <td align="right">63.2</td><td align="right">60.0</td><td align="right">+3.2</td> </tr> <tr> <td>Claude Sonnet 4.6</td> <td align="right">90.0</td><td align="right">77.5</td><td align="right">+12.5</td> <td align="right">35.9</td><td align="right">42.6</td><td align="right">-6.7</td> <td align="right">57.5</td><td align="right">56.6</td><td align="right">+0.9</td> </tr> <tr><td colspan="10">Open-source models</td></tr> <tr> <td>EHR-R1-72B</td> <td align="right">--</td><td align="right">67.1</td><td align="right">--</td> <td align="right">--</td><td align="right">45.2</td><td align="right">--</td> <td align="right">--</td><td align="right">53.9</td><td align="right">--</td> </tr> <tr> <td>GLM-4.7</td> <td align="right">75.1</td><td align="right">70.4</td><td align="right">+4.7</td> <td align="right">23.1</td><td align="right">38.6</td><td align="right">-15.5</td> <td align="right">43.9</td><td align="right">51.3</td><td align="right">-7.4</td> </tr> <tr> <td>Qwen3.5-35B-A3B</td> <td align="right">84.4</td><td align="right">73.6</td><td align="right">+10.8</td> <td align="right">22.0</td><td align="right">29.0</td><td align="right">-7.0</td> <td align="right">47.0</td><td align="right">46.8</td><td align="right">+0.1</td> </tr> <tr> <td>Gemma-4-26B-A4B-it</td> <td align="right">83.5</td><td align="right">78.6</td><td align="right">+4.9</td> <td align="right">17.3</td><td align="right">27.8</td><td align="right">-10.5</td> <td align="right">43.8</td><td align="right">48.1</td><td align="right">-4.3</td> </tr> <tr> <td>MiniMax M2.5</td> <td align="right">86.7</td><td align="right">68.4</td><td align="right">+18.3</td> <td align="right">21.0</td><td align="right">26.3</td><td align="right">-5.3</td> <td align="right">47.3</td><td align="right">43.1</td><td align="right">+4.2</td> </tr> <tr> <td>Kimi K2.5</td> <td align="right">65.0</td><td align="right">79.9</td><td align="right">-14.9</td> <td align="right">19.8</td><td align="right">28.8</td><td align="right">-9.0</td> <td align="right">37.9</td><td align="right">49.2</td><td align="right">-11.3</td> </tr> <tr> <td>Qwen3-VL-235B</td> <td align="right">67.9</td><td align="right">71.0</td><td align="right">-3.1</td> <td align="right">19.1</td><td align="right">33.4</td><td align="right">-14.3</td> <td align="right">38.6</td><td align="right">48.4</td><td align="right">-9.8</td> </tr> <tr> <td>gpt-oss-120b</td> <td align="right">75.4</td><td align="right">74.0</td><td align="right">+1.4</td> <td align="right">16.6</td><td align="right">22.3</td><td align="right">-5.7</td> <td align="right">40.1</td><td align="right">43.0</td><td align="right">-2.9</td> </tr> <tr> <td>MedGemma-27B-it</td> <td align="right">--</td><td align="right">65.0</td><td align="right">--</td> <td align="right">--</td><td align="right">25.2</td><td align="right">--</td> <td align="right">--</td><td align="right">41.1</td><td align="right">--</td> </tr> <tr> <td>EHR-R1-8B</td> <td align="right">--</td><td align="right">64.0</td><td align="right">--</td> <td align="right">--</td><td align="right">23.4</td><td align="right">--</td> <td align="right">--</td><td align="right">39.7</td><td align="right">--</td> </tr> </tbody> </table> ### Multimodal tasks <table> <thead> <tr> <th>Model</th> <th>Method</th> <th>CXR: finding presence</th> <th>CXR: finding enumeration</th> <th>CXR: change comparison</th> <th>Mortality (24 h)</th> <th>Inpatient mortality</th> <th>Phenotype (CCS groups)</th> <th>Multimodal overall</th> </tr> </thead> <tbody> <tr> <td rowspan="3">Claude Opus 4.6</td> <td>ClinSeekAgent</td> <td align="right">78.3</td><td align="right">43.6</td><td align="right">54.8</td><td align="right">92.0</td><td align="right">74.4</td><td align="right">45.5</td><td align="right">62.6</td> </tr> <tr> <td>Curated Input</td> <td align="right">55.2</td><td align="right">31.6</td><td align="right">38.0</td><td align="right">93.6</td><td align="right">69.6</td><td align="right">11.5</td><td align="right">47.5</td> </tr> <tr> <td>Δ</td> <td align="right">+23.2</td><td align="right">+12.0</td><td align="right">+16.8</td><td align="right">-1.6</td><td align="right">+4.8</td><td align="right">+34.0</td><td align="right">+15.1</td> </tr> <tr> <td rowspan="3">Claude Sonnet 4.6</td> <td>ClinSeekAgent</td> <td align="right">79.5</td><td align="right">41.3</td><td align="right">51.5</td><td align="right">64.0</td><td align="right">68.8</td><td align="right">26.1</td><td align="right">54.9</td> </tr> <tr> <td>Curated Input</td> <td align="right">64.8</td><td align="right">29.7</td><td align="right">34.7</td><td align="right">90.4</td><td align="right">70.4</td><td align="right">13.8</td><td align="right">48.0</td> </tr> <tr> <td>Δ</td> <td align="right">+14.7</td><td align="right">+11.6</td><td align="right">+16.8</td><td align="right">-26.4</td><td align="right">-1.6</td><td align="right">+12.3</td><td align="right">+6.9</td> </tr> <tr> <td rowspan="3">Qwen3.5-35B-A3B</td> <td>ClinSeekAgent</td> <td align="right">73.8</td><td align="right">34.2</td><td align="right">44.4</td><td align="right">91.2</td><td align="right">74.4</td><td align="right">0.3</td><td align="right">51.7</td> </tr> <tr> <td>Curated Input</td> <td align="right">59.1</td><td align="right">34.1</td><td align="right">30.7</td><td align="right">90.4</td><td align="right">81.6</td><td align="right">0.5</td><td align="right">46.9</td> </tr> <tr> <td>Δ</td> <td align="right">+14.7</td><td align="right">+0.2</td><td align="right">+13.7</td><td align="right">+0.8</td><td align="right">-7.2</td><td align="right">-0.2</td><td align="right">+4.8</td> </tr> <tr> <td rowspan="3">Kimi K2.5</td> <td>ClinSeekAgent</td> <td align="right">61.4</td><td align="right">34.9</td><td align="right">43.8</td><td align="right">71.2</td><td align="right">62.4</td><td align="right">12.3</td><td align="right">46.9</td> </tr> <tr> <td>Curated Input</td> <td align="right">56.3</td><td align="right">24.7</td><td align="right">35.0</td><td align="right">91.2</td><td align="right">87.2</td><td align="right">12.4</td><td align="right">47.5</td> </tr> <tr> <td>Δ</td> <td align="right">+5.1</td><td align="right">+10.2</td><td align="right">+8.8</td><td align="right">-20.0</td><td align="right">-24.8</td><td align="right">-0.1</td><td align="right">-0.6</td> </tr> <tr> <td rowspan="3">Qwen3-VL-235B</td> <td>ClinSeekAgent</td> <td align="right">70.4</td><td align="right">35.7</td><td align="right">47.8</td><td align="right">79.2</td><td align="right">61.6</td><td align="right">6.0</td><td align="right">49.8</td> </tr> <tr> <td>Curated Input</td> <td align="right">60.3</td><td align="right">21.1</td><td align="right">32.8</td><td align="right">87.2</td><td align="right">72.8</td><td align="right">6.6</td><td align="right">43.9</td> </tr> <tr> <td>Δ</td> <td align="right">+10.1</td><td align="right">+14.6</td><td align="right">+15.0</td><td align="right">-8.0</td><td align="right">-11.2</td><td align="right">-0.6</td><td align="right">+5.9</td> </tr> <tr> <td rowspan="3">Gemma-4-26B-A4B-it</td> <td>ClinSeekAgent</td> <td align="right">78.9</td><td align="right">21.6</td><td align="right">38.4</td><td align="right">65.6</td><td align="right">71.2</td><td align="right">0.4</td><td align="right">44.9</td> </tr> <tr> <td>Curated Input</td> <td align="right">56.9</td><td align="right">21.4</td><td align="right">25.4</td><td align="right">79.2</td><td align="right">60.0</td><td align="right">0.0</td><td align="right">38.2</td> </tr> <tr> <td>Δ</td> <td align="right">+22.0</td><td align="right">+0.2</td><td align="right">+13.0</td><td align="right">-13.6</td><td align="right">+11.2</td><td align="right">+0.4</td><td align="right">+6.7</td> </tr> </tbody> </table> These results show that active evidence acquisition is most helpful when the answer depends on sparse, longitudinal, or multimodal signals that may be missed by fixed curated inputs. The gains are especially clear for risk prediction in the text-only split and for CXR-related multimodal tasks, where models can combine EHR retrieval, image-tool outputs, and task-relevant medical knowledge. ## Files ```text ClinSeek-Bench/ +-- README.md +-- ClinSeek-Bench_text.json +-- ClinSeek-Bench_multimodal.jsonl ``` The paths above are the released metadata paths in this dataset repository. If you clone this dataset next to the ClinSeekAgent repository, reference the metadata as: ```text ClinSeek-Bench/ClinSeek-Bench_text.json ClinSeek-Bench/ClinSeek-Bench_multimodal.jsonl ``` If you use ClinSeekAgent's default local data layout, you may copy or symlink the text manifest to `data/text/ClinSeek-Bench_text.json`. That location is a local workspace convenience, not a released metadata path. ## Reconstructing The Full Benchmark Reconstruction code and detailed instructions are maintained in the ClinSeekAgent GitHub repository: - Code: https://github.com/UCSC-VLAA/ClinSeekAgent - Text data preparation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_data_prepare.md - Automated Evidence-Seeking text evaluation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_evaluation.md - Curated Input text evaluation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_curated_input_evaluation.md - Multimodal data preparation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_multimodal_data_prepare.md At a high level: ```bash git clone https://huggingface.co/datasets/UCSC-VLAA/ClinSeek-Bench git clone https://github.com/UCSC-VLAA/ClinSeekAgent.git ``` Then obtain the required official source datasets, prepare local paths, and follow the text and multimodal reconstruction guides above. ## Required Source Data Download clinical source data only from official sources under your own credentialed access and data-use agreements. | Dataset or source | Official source | Used for | | --- | --- | --- | | MIMIC-IV | https://physionet.org/content/mimiciv/ | structured EHR tables | | MIMIC-IV-Note | https://physionet.org/content/mimic-iv-note/ | discharge and radiology notes | | MIMIC-IV-ED | https://physionet.org/content/mimic-iv-ed/ | emergency department tables | | MIMIC-CXR | https://physionet.org/content/mimic-cxr/ | CXR metadata and report text | | MIMIC-CXR-JPG | https://physionet.org/content/mimic-cxr-jpg/ | linked chest X-ray JPG files | | EHRXQA | https://physionet.org/content/ehrxqa/ | source-aligned CXR question-answering rows | | MedMod | https://github.com/nyuad-cai/MedMod | source-aligned ICU/CXR prediction rows | | EHR-R1 / EHR-Bench | https://github.com/MAGIC-AI4Med/EHR-R1 | source benchmark for text-only EHR tasks | ## Validation After cloning ClinSeekAgent, validate the released multimodal metadata without requiring protected assets: ```bash python ClinSeekAgent/scripts/data_build/validate_multimodal_release.py \ --bench-root ClinSeek-Bench \ --input ClinSeek-Bench/ClinSeek-Bench_multimodal.jsonl \ --manifest-only ``` Expected released metadata counts: - 1,800 text-only rows; - 45 text-only EHR subtasks; - 1,563 unique text-only patients; - 989 multimodal rows; - 497 EHRXQA-derived multimodal rows; - 492 MedMod-derived multimodal rows; - 350 unique EHRXQA linked CXR JPG paths; - 477 unique MedMod linked CXR JPG paths; - 356 unique EHRXQA linked CXR report paths. After local reconstruction, the same validation script can also check the materialized runtime assets, including patient DBs, image paths, and report paths. See the ClinSeekAgent reconstruction documentation for the full commands. ## Evaluation Evaluation code and instructions are maintained in the ClinSeekAgent repository: https://github.com/UCSC-VLAA/ClinSeekAgent. After reconstructing the protected runtime assets locally, follow the ClinSeekAgent documentation for Automated Evidence-Seeking evaluation, Curated Input evaluation, and scoring. ## Responsible Use ClinSeek-Bench is for research on clinical evidence seeking. It is not a medical device and must not be used for clinical diagnosis, treatment, triage, or patient management without separate validation, governance, and regulatory review.

提供机构：

maas

创建时间：

2026-05-21

ClinSeek-Bench

资源简介：

相关数据集