jdh-algo/MedDocBench

---
license: apache-2.0
configs:
- config_name: v20251015-bytes
data_files:
- split: GMD_complexQA
path: "GMD_complexQA/publish-20251015.bytes.parquet"
- split: GMD_simpleQA
path: "GMD_simpleQA/publish-20251015.bytes.parquet"
- split: LTR_abnormalityQA
path: "LTR_abnormalityQA/publish-20251015.bytes.parquet"
- split: LTR_fullparsing
path: "LTR_fullparsing/publish-20251015.bytes.parquet"
- split: LTR_simpleQA
path: "LTR_simpleQA/publish-20251015.bytes.parquet"
---

## MedDocBench

A compact benchmark of text‑rich medical document understanding covering routine, patient-uploaded artifacts from online consultations, spanning laboratory test reports (LTR) and general medical documents (GMD).

## Configuration and splits

- **Configuration**: `v20251015-bytes`
- **Available splits**:
- **LTR_fullparsing**: 100
- **LTR_simpleQA**: 200
- **LTR_abnormalityQA (complex QA)**: 100
- **GMD_simpleQA**: 100
- **GMD_complexQA**: 100
- **Total**: 600 QA pairs

### Data formats

- `publish-20251015.bytes.parquet`: images stored as base64-encoded bytes with relative paths.
- `publish-20251015.parquet`: images referenced via relative paths (no embedded bytes).
- `tsv` files: images referenced via relative paths, for direct evaluation with EvalScope/VLMEvalKit.

## Statistics

**Angle distribution (all tasks)**: 0°/90°/180°/270° = 25%/25%/25%/25%

### LTR image distribution

| Status | Capture method | Count |
|---|---|---:|
| Normal | Mobile | 4 |
| Normal | Paper | 16 |
| Abnormal | Mobile | 16 |
| Abnormal | Paper | 64 |

### GMD document type distribution

- **GMD_simpleQA**

| Doc type | Count |
|---|---:|
| Laboratory Test Reports | 30 |
| Medication Packages | 19 |
| Imaging Reports | 11 |
| Outpatient Encounter Notes | 9 |
| Other Diagnostic Reports | 9 |
| Inpatient Records | 7 |
| Prescriptions (Western Medicine) | 6 |
| Other Clinical Documents | 4 |
| Prescriptions (TCM) | 4 |
| Health Records | 1 |

- **GMD_complexQA**

| Doc type | Count |
|---|---:|
| Medication Packages | 24 |
| Imaging Reports | 15 |
| Other Diagnostic Reports | 13 |
| Laboratory Test Reports | 13 |
| Prescriptions (Western Medicine) | 10 |
| Outpatient Encounter Notes | 7 |
| Inpatient Records | 5 |
| Other Clinical Documents | 4 |
| Prescriptions (TCM) | 4 |
| Other text-rich images | 3 |
| Health Records | 2 |

## Load with datasets

```python
from datasets import load_dataset

# Load the HuggingFace dataset and pick the configuration
ds = load_dataset("<ORG_OR_USER>/MedDocBench", "v20251015-bytes")

# Access individual splits by name
gmd_simple = ds["GMD_simpleQA"]
ltr_full = ds["LTR_fullparsing"]
```

## Evaluation

Based on EvalScope with a VLMEvalKit backend. See the EvalScope documentation: [evalscope.readthedocs.io](https://evalscope.readthedocs.io/en/latest/index.html).

```bash
# 1) Create env (Python 3.10 recommended)
conda create -n evalscope python=3.10 -y
conda activate evalscope

# 2) Install EvalScope with VLMEvalKit extras
pip install "evalscope[vlmeval]"

# 3) Point EvalScope to your TSVs used by benchmark_vlmevalkit
# Create or edit the .env file under the installed package, e.g.:
# $CONDA_PREFIX/lib/python3.10/site-packages/.env
# and add:
# LMUData=/path/to/tsv_files

# 4) Run evaluation from the benchmark folder
cd MedDocBench/evaluation
python eval.py --config eval_config.yaml
```

## Citation

If you use this benchmark, please cite:

**Citrus‑V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning**
[arXiv:2509.19090](https://arxiv.org/abs/2509.19090)

## License
license: apache-2.0