biomed-VQA-benchmark

# Adapting Multimodal Large Language Models to Domains via Post-Training (EMNLP 2025)

This repos contains the **biomedical visual instruction tasks for evaluating MLLMs** in our paper: [On Domain-Specific Post-Training for Multimodal Large Language Models](https://huggingface.co/papers/2411.19930).

The main project page is: [Adapt-MLLM-to-Domains](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains)

## 1. Download Data
You can load datasets using the `datasets` library:
```python
from datasets import load_dataset

# Choose the task name from the list of available tasks
task_name = 'SLAKE' # Options: 'SLAKE', 'VQA_RAD', 'PathVQA', 'PMC-VQA'

# Load the dataset for the chosen task
data = load_dataset('AdaptLLM/biomed-VQA-benchmark', task_name, split='test')

print(list(data)[0])
```

## 2. Evaluate Any MLLM Compatible with vLLM on the BioMed Benchmarks

We provide a guide to directly evaluate MLLMs such as LLaVA-v1.6 ([open-source version](https://huggingface.co/Lin-Chen/open-llava-next-llama3-8b)), Qwen2-VL-Instruct, and Llama-3.2-Vision-Instruct.
To evaluate other MLLMs, refer to [this guide](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py) for modifying the `BaseTask` class in the [vllm_inference/utils/task.py](https://github.com/bigai-ai/QA-Synthesizer/blob/main/vllm_inference/utils/task.py) file.
Feel free reach out to us for assistance!

**The dataset loading script is embedded in the inference code, so you can directly run the following commands to evaluate MLLMs.**

### 1) Setup

Install vLLM using `pip` or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source).

As recommended in the official vLLM documentation, install vLLM in a **fresh new** conda environment:

```bash
conda create -n vllm python=3.10 -y
conda activate vllm
pip install vllm # Ensure vllm>=0.6.2 for compatibility with Llama-3.2. If Llama-3.2 is not used, vllm==0.6.1 is sufficient.
```

Clone the repository and navigate to the inference directory:

```bash
git clone https://github.com/bigai-ai/QA-Synthesizer.git
cd QA-Synthesizer/vllm_inference
RESULTS_DIR=./eval_results # Directory for saving evaluation scores
```

### 2) Evaluate

Run the following commands:

```bash
# Specify the domain: choose from ['med', 'PMC_VQA', 'VQA_RAD', 'SLAKE', 'PathVQA']
# 'med' runs inference on all biomedicine tasks; others run on a single task
DOMAIN='med'

# Specify the model type: choose from ['llava', 'qwen2_vl', 'mllama']
# For LLaVA-v1.6, Qwen2-VL, and Llama-3.2-Vision-Instruct, respectively.
MODEL_TYPE='qwen2_vl'

# Set the model repository ID on Hugging Face. Examples:
# "Qwen/Qwen2-VL-2B-Instruct", "AdaptLLM/biomed-Qwen2-VL-2B-Instruct" for MLLMs based on Qwen2-VL-Instruct.
# "meta-llama/Llama-3.2-11B-Vision-Instruct", "AdaptLLM/biomed-Llama-3.2-11B-Vision-Instruct" for MLLMs based on Llama-3.2-Vision-Instruct.
# "AdaptLLM/biomed-LLaVA-NeXT-Llama3-8B" for MLLMs based on LLaVA-v1.6.
MODEL=AdaptLLM/biomed-Qwen2-VL-2B-Instruct

# Set the directory for saving model prediction outputs:
OUTPUT_DIR=./output/AdaMLLM-med-Qwen-2B_${DOMAIN}

# Run inference with data parallelism; adjust CUDA devices as needed:
CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' bash run_inference.sh ${MODEL} ${DOMAIN} ${MODEL_TYPE} ${OUTPUT_DIR} ${RESULTS_DIR}
```

Detailed scripts to reproduce our results are in [Evaluation.md](https://github.com/bigai-ai/QA-Synthesizer/blob/main/docs/Evaluation.md)

### 3) Results
The evaluation results are stored in `./eval_results`, and the model prediction outputs are in `./output`.

## Citation
If you find our work helpful, please cite us.

[Adapt MLLM to Domains](https://huggingface.co/papers/2411.19930) (EMNLP 2025 Findings)
```bibtex
@article{adamllm,
title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
author={Cheng, Daixuan and Huang, Shaohan and Zhu, Ziyu and Zhang, Xintong and Zhao, Wayne Xin and Luan, Zhongzhi and Dai, Bo and Zhang, Zhenliang},
journal={arXiv preprint arXiv:2411.19930},
year={2024}
}
```

[Adapt LLM to Domains](https://huggingface.co/papers/2309.09530) (ICLR 2024)
```bibtex
@inproceedings{
adaptllm,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}
```

# 基于后训练将多模态大语言模型适配至特定领域（EMNLP 2025）

本仓库收录了我们论文《面向多模态大语言模型的领域专属后训练研究》（[https://huggingface.co/papers/2411.19930](https://huggingface.co/papers/2411.19930)）中用于评估多模态大语言模型（Multimodal Large Language Model, MLLM）的**生物医学视觉指令任务**。

主项目页面为：[Adapt-MLLM-to-Domains](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains)

## 1. 下载数据
您可通过`datasets`库加载目标数据集：
python
from datasets import load_dataset

# 从可用任务列表中选取任务名称
task_name = 'SLAKE' # 可选任务包括：'SLAKE'、'VQA_RAD'、'PathVQA'、'PMC-VQA'

# 加载所选任务的测试集
data = load_dataset('AdaptLLM/biomed-VQA-benchmark', task_name, split='test')

print(list(data)[0])

## 2. 在生物医学基准测试集上评估兼容vLLM的任意多模态大语言模型
我们提供了直接评估各类多模态大语言模型的实操指南，涵盖LLaVA-v1.6（[开源版本](https://huggingface.co/Lin-Chen/open-llava-next-llama3-8b)）、Qwen2-VL-Instruct及Llama-3.2-Vision-Instruct等模型。
若需评估其他多模态大语言模型，请参考[官方指南](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py)，修改[vllm_inference/utils/task.py](https://github.com/bigai-ai/QA-Synthesizer/blob/main/vllm_inference/utils/task.py)中的`BaseTask`类。
欢迎随时联系我们获取技术支持！

**数据集加载脚本已内嵌至推理代码中，您可直接运行以下命令完成多模态大语言模型的评估。**

### 1) 环境配置
通过`pip`或[源码编译方式](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source)安装vLLM。

根据vLLM官方文档建议，请在**全新的**conda虚拟环境中完成安装：
bash
conda create -n vllm python=3.10 -y
conda activate vllm
pip install vllm # 若需兼容Llama-3.2，请确保vllm版本≥0.6.2；若不使用Llama-3.2，vllm==0.6.1即可满足需求。

克隆本仓库并进入推理目录：
bash
git clone https://github.com/bigai-ai/QA-Synthesizer.git
cd QA-Synthesizer/vllm_inference
RESULTS_DIR=./eval_results # 评估分数的保存目录

### 2) 模型评估
执行以下命令启动评估：
bash
# 指定评估领域：可选值为['med', 'PMC_VQA', 'VQA_RAD', 'SLAKE', 'PathVQA']
# 'med' 表示在全部生物医学任务上运行推理；其余选项仅针对单个任务
DOMAIN='med'

# 指定模型类型：可选值为['llava', 'qwen2_vl', 'mllama']
# 分别对应LLaVA-v1.6、Qwen2-VL及Llama-3.2-Vision-Instruct模型
MODEL_TYPE='qwen2_vl'

# 设置Hugging Face平台上的模型仓库ID，示例如下：
# 基于Qwen2-VL-Instruct的模型："Qwen/Qwen2-VL-2B-Instruct"、"AdaptLLM/biomed-Qwen2-VL-2B-Instruct"
# 基于Llama-3.2-Vision-Instruct的模型："meta-llama/Llama-3.2-11B-Vision-Instruct"、"AdaptLLM/biomed-Llama-3.2-11B-Vision-Instruct"
# 基于LLaVA-v1.6的模型："AdaptLLM/biomed-LLaVA-NeXT-Llama3-8B"
MODEL=AdaptLLM/biomed-Qwen2-VL-2B-Instruct

# 设置模型预测结果的保存目录：
OUTPUT_DIR=./output/AdaMLLM-med-Qwen-2B_${DOMAIN}

# 启用数据并行推理；请根据实际硬件情况调整CUDA设备编号：
CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' bash run_inference.sh ${MODEL} ${DOMAIN} ${MODEL_TYPE} ${OUTPUT_DIR} ${RESULTS_DIR}

用于复现本文实验结果的详细脚本可参见[Evaluation.md](https://github.com/bigai-ai/QA-Synthesizer/blob/main/docs/Evaluation.md)

### 3) 评估结果
评估结果将存储于`./eval_results`目录，模型的预测输出将保存至`./output`目录。

## 引用
若本工作对您的研究有所帮助，请引用以下论文：

[Adapt MLLM to Domains](https://huggingface.co/papers/2411.19930)（EMNLP 2025 发现论文）
bibtex
@article{adamllm,
title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
author={Cheng, Daixuan and Huang, Shaohan and Zhu, Ziyu and Zhang, Xintong and Zhao, Wayne Xin and Luan, Zhongzhi and Dai, Bo and Zhang, Zhenliang},
journal={arXiv preprint arXiv:2411.19930},
year={2024}
}

[Adapt LLM to Domains](https://huggingface.co/papers/2309.09530)（ICLR 2024）
bibtex
@inproceedings{
adaptllm,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}