med_knowledge_prob

# Adapting LLMs to Domains via Continual Pre-Training (ICLR 2024)
This repo contains the **Biomedicine Knowledge Probing dataset** used in our paper [Adapting Large Language Models via Reading Comprehension](https://huggingface.co/papers/2309.09530).

We explore **continued pre-training on domain-specific corpora** for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to **transform large-scale pre-training corpora into reading comprehension texts**, consistently improving prompting performance across tasks in biomedicine, finance, and law domains. **Our 7B model competes with much larger domain-specific models like BloombergGPT-50B**.

### [2024/11/29] 🤗 Introduce the multimodal version of AdaptLLM at [AdaMLLM](https://huggingface.co/papers/2411.19930), for adapting MLLMs to domains 🤗

**************************** **Updates** ****************************
* 2024/11/29: Released [AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains) for adapting MLLMs to domains
* 2024/9/20: Our [research paper for Instruction-Pretrain](https://huggingface.co/papers/2406.14491) has been accepted by EMNLP 2024
* 2024/8/29: Updated [guidelines](https://huggingface.co/datasets/AdaptLLM/finance-tasks) on evaluating any 🤗Huggingface models on the domain-specific tasks
* 2024/6/22: Released the [benchmarking code](https://github.com/microsoft/LMOps/tree/main/adaptllm)
* 2024/6/21: Released the general version of AdaptLLM at [Instruction-Pretrain](https://huggingface.co/instruction-pretrain)
* 2024/4/2: Released the [raw data splits (train and test)](https://huggingface.co/datasets/AdaptLLM/ConvFinQA) of all the evaluation datasets
* 2024/1/16: Our [research paper for AdaptLLM](https://huggingface.co/papers/2309.09530) has been accepted by ICLR 2024
* 2023/12/19: Released our [13B base models](https://huggingface.co/AdaptLLM/law-LLM-13B) developed from LLaMA-1-13B
* 2023/12/8: Released our [chat models](https://huggingface.co/AdaptLLM/law-chat) developed from LLaMA-2-Chat-7B
* 2023/9/18: Released our [paper](https://huggingface.co/papers/2309.09530), [code](https://github.com/microsoft/LMOps), [data](https://huggingface.co/datasets/AdaptLLM/law-tasks), and [base models](https://huggingface.co/AdaptLLM/law-LLM) developed from LLaMA-1-7B

## 1. Domain-Specific Models
### LLaMA-1-7B
In our paper, we develop three domain-specific models from LLaMA-1-7B, which are also available in Huggingface: [Biomedicine-LLM](https://huggingface.co/AdaptLLM/medicine-LLM), [Finance-LLM](https://huggingface.co/AdaptLLM/finance-LLM) and [Law-LLM](https://huggingface.co/AdaptLLM/law-LLM), the performances of our AdaptLLM compared to other domain-specific LLMs are:

<p align='center'>
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">
</p>

### LLaMA-1-13B
Moreover, we scale up our base model to LLaMA-1-13B to see if **our method is similarly effective for larger-scale models**, and the results are consistently positive too: [Biomedicine-LLM-13B](https://huggingface.co/AdaptLLM/medicine-LLM-13B), [Finance-LLM-13B](https://huggingface.co/AdaptLLM/finance-LLM-13B) and [Law-LLM-13B](https://huggingface.co/AdaptLLM/law-LLM-13B).

### LLaMA-2-Chat
Our method is also effective for aligned models! LLaMA-2-Chat requires a [specific data format](https://huggingface.co/blog/llama2#how-to-prompt-llama-2), and our **reading comprehension can perfectly fit the data format** by transforming the reading comprehension into a multi-turn conversation. We have also open-sourced chat models in different domains: [Biomedicine-Chat](https://huggingface.co/AdaptLLM/medicine-chat), [Finance-Chat](https://huggingface.co/AdaptLLM/finance-chat) and [Law-Chat](https://huggingface.co/AdaptLLM/law-chat).

### LLaMA-3-8B (💡New!)
In our recent research on [Instruction-Pretrain](https://huggingface.co/papers/2406.14491), we developed a context-based instruction synthesizer to augment the raw corpora with instruction-response pairs, **enabling Llama3-8B to be comparable to or even outperform Llama3-70B**: [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B), [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B).

## 2. Domain-Specific Tasks

### Pre-templatized Testing Splits
To easily reproduce our prompting results, we have uploaded the filled-in zero/few-shot input instructions and output completions of the test each domain-specific task: [biomedicine-tasks](https://huggingface.co/datasets/AdaptLLM/medicine-tasks), [finance-tasks](https://huggingface.co/datasets/AdaptLLM/finance-tasks), and [law-tasks](https://huggingface.co/datasets/AdaptLLM/law-tasks).

Note: those filled-in instructions are specifically tailored for models before alignment and do NOT fit for the specific data format required for chat models.

### Evaluating Any Huggingface LMs on Domain-Specific Tasks (💡New!)
You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).

1). **Set Up Dependencies**
```bash
git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt
```

2). **Evaluate the Model**
```bash
# Select the domain from ['biomedicine', 'finance', 'law']
DOMAIN='biomedicine'

# Specify any Huggingface model name (Not applicable to chat models)
MODEL='instruction-pretrain/medicine-Llama3-8B'

# Model parallelization:
# - Set MODEL_PARALLEL=False if the model fits on a single GPU.
# We observe that LMs smaller than 10B always meet this requirement.
# - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU.
MODEL_PARALLEL=False

# Choose the number of GPUs from [1, 2, 4, 8]
N_GPU=1

# Whether to add a BOS token at the beginning of the prompt input:
# - Set to False for AdaptLLM.
# - Set to True for instruction-pretrain models.
# If unsure, we recommend setting it to False, as this is suitable for most LMs.
add_bos_token=True

# Run the evaluation script
bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}
```

### Raw Datasets
We have also uploaded the raw training and testing splits, for facilitating fine-tuning or other usages: [ChemProt](https://huggingface.co/datasets/AdaptLLM/ChemProt), [RCT](https://huggingface.co/datasets/AdaptLLM/RCT), [ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA), [FiQA_SA](https://huggingface.co/datasets/AdaptLLM/FiQA_SA), [Headline](https://huggingface.co/datasets/AdaptLLM/Headline), [NER](https://huggingface.co/datasets/AdaptLLM/NER), [FPB](https://huggingface.co/datasets/AdaptLLM/FPB)

### Domain Knowledge Probing
Our pre-processed knowledge probing datasets are available at: [med_knowledge_prob](https://huggingface.co/datasets/AdaptLLM/med_knowledge_prob) and [law_knowledge_prob](https://huggingface.co/datasets/AdaptLLM/law_knowledge_prob)

## Citation
If you find our work helpful, please cite us:
```bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}
```

and the original dataset:
```bibtex
@inproceedings{MedMCQA,
author = {Ankit Pal and
Logesh Kumar Umapathi and
Malaikannan Sankarasubbu},
title = {MedMCQA: {A} Large-scale Multi-Subject Multi-Choice Dataset for Medical
domain Question Answering},
booktitle = {{CHIL}},
series = {Proceedings of Machine Learning Research},
volume = {174},
pages = {248--260},
publisher = {{PMLR}},
year = {2022}
}
```

# 通过持续预训练实现大语言模型（LLM）的领域适配（ICLR 2024）
本仓库包含我们发表于论文《通过阅读理解适配大语言模型》（https://huggingface.co/papers/2309.09530）中使用的**生物医药知识探测数据集（Biomedicine Knowledge Probing dataset）**。

我们针对大语言模型（LLM）探索了**基于领域专属语料的持续预训练**方案。尽管该方法能够为大语言模型注入领域知识，但会显著损害其问答任务的提示学习能力。受人类通过阅读理解进行学习的启发，我们提出了一种简单方法，可**将大规模预训练语料转换为阅读理解文本**，在生物医药、金融与法律领域的各类任务中持续提升提示学习性能。我们的7B参数模型可与BloombergGPT-50B等超大规模领域专属模型相媲美。

[2024/11/29] 🤗 推出AdaptLLM的多模态版本AdaMLLM（https://huggingface.co/papers/2411.19930），用于多模态大语言模型（MLLM）的领域适配 🤗

**************************** **更新日志** ****************************
* 2024/11/29：发布用于多模态大语言模型领域适配的AdaMLLM（https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains）
* 2024/9/20：我们的[指令预训练研究论文](https://huggingface.co/papers/2406.14491)已被EMNLP 2024收录
* 2024/8/29：更新了[评估指南](https://huggingface.co/datasets/AdaptLLM/finance-tasks)，用于在领域专属任务上评估任意🤗Huggingface模型
* 2024/6/22：发布[基准测试代码](https://github.com/microsoft/LMOps/tree/main/adaptllm)
* 2024/6/21：在[指令预训练（Instruction-Pretrain）](https://huggingface.co/instruction-pretrain)平台发布AdaptLLM的通用版本
* 2024/4/2：发布所有评估数据集的[原始数据拆分（训练集与测试集）](https://huggingface.co/datasets/AdaptLLM/ConvFinQA)
* 2024/1/16：我们的[AdaptLLM研究论文](https://huggingface.co/papers/2309.09530)已被ICLR 2024收录
* 2023/12/19：发布基于LLaMA-1-13B开发的[13B参数基础模型](https://huggingface.co/AdaptLLM/law-LLM-13B)
* 2023/12/8：发布基于LLaMA-2-Chat-7B开发的[对话模型](https://huggingface.co/AdaptLLM/law-chat)
* 2023/9/18：发布基于LLaMA-1-7B开发的[研究论文](https://huggingface.co/papers/2309.09530)、[代码](https://github.com/microsoft/LMOps)、[数据集](https://huggingface.co/datasets/AdaptLLM/law-tasks)以及[基础模型](https://huggingface.co/AdaptLLM/law-LLM)

## 1. 领域专属模型
### LLaMA-1-7B
在本研究中，我们基于LLaMA-1-7B开发了三款领域专属模型，现已在Huggingface平台上架：生物医药大语言模型（Biomedicine-LLM，https://huggingface.co/AdaptLLM/medicine-LLM）、金融大语言模型（Finance-LLM，https://huggingface.co/AdaptLLM/finance-LLM）以及法律大语言模型（Law-LLM，https://huggingface.co/AdaptLLM/law-LLM）。我们的AdaptLLM与其他领域专属大语言模型的性能对比如下：

<p align='center'>
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">
</p>

### LLaMA-1-13B
此外，我们将基础模型扩展至LLaMA-1-13B，以验证**我们的方法在更大规模模型上是否同样有效**，实验结果同样表现优异：生物医药大语言模型-13B（Biomedicine-LLM-13B，https://huggingface.co/AdaptLLM/medicine-LLM-13B）、金融大语言模型-13B（Finance-LLM-13B，https://huggingface.co/AdaptLLM/finance-LLM-13B）以及法律大语言模型-13B（Law-LLM-13B，https://huggingface.co/AdaptLLM/law-LLM-13B）。

### LLaMA-2-Chat
我们的方法对对齐后的模型同样有效！LLaMA-2-Chat要求使用[特定的数据格式](https://huggingface.co/blog/llama2#how-to-prompt-llama-2)，而我们的**阅读理解文本可通过转换为多轮对话完美适配该格式**。我们还开源了不同领域的对话模型：生物医药对话模型（Biomedicine-Chat，https://huggingface.co/AdaptLLM/medicine-chat）、金融对话模型（Finance-Chat，https://huggingface.co/AdaptLLM/finance-chat）以及法律对话模型（Law-Chat，https://huggingface.co/AdaptLLM/law-chat）。

### LLaMA-3-8B（💡新增！）
在我们近期关于[指令预训练（Instruction-Pretrain，https://huggingface.co/papers/2406.14491）]的研究中，我们开发了一种基于上下文的指令合成器，用于为原始语料补充指令-响应对，**使Llama3-8B可媲美甚至超越Llama3-70B**：金融-Llama3-8B（Finance-Llama3-8B，https://huggingface.co/instruction-pretrain/finance-Llama3-8B）、生物医药-Llama3-8B（Biomedicine-Llama3-8B，https://huggingface.co/instruction-pretrain/medicine-Llama3-8B）。

## 2. 领域专属任务

### 预模板化测试拆分
为便于复现我们的提示学习实验结果，我们上传了各领域专属任务测试集的填充好的零样本/少样本输入指令与输出结果：生物医药任务集（biomedicine-tasks，https://huggingface.co/datasets/AdaptLLM/medicine-tasks）、金融任务集（finance-tasks，https://huggingface.co/datasets/AdaptLLM/finance-tasks）以及法律任务集（law-tasks，https://huggingface.co/datasets/AdaptLLM/law-tasks）。

注意：此类填充指令专为对齐前的模型设计，不适用于对话模型要求的特定数据格式。

### 在领域专属任务上评估任意Huggingface大语言模型（💡新增！）
您可使用以下脚本复现我们的实验结果，并在领域专属任务上评估其他任意Huggingface模型。注意：该脚本不适用于需要特定提示模板的模型（如Llama2-chat、Llama3-Instruct）。

1). **设置依赖环境**
bash
git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt

2). **评估模型**
bash
# 从['biomedicine', 'finance', 'law']中选择领域
DOMAIN='biomedicine'

# 指定任意Huggingface模型名称（不适用于对话模型）
MODEL='instruction-pretrain/medicine-Llama3-8B'

# 模型并行设置：
# - 若模型可单张GPU运行，则将MODEL_PARALLEL设为False。
# 我们发现参数小于10B的大语言模型均可满足该条件。
# - 若模型体积过大导致单张GPU显存不足，则将MODEL_PARALLEL设为True。
MODEL_PARALLEL=False

# 选择GPU数量，可选[1, 2, 4, 8]
N_GPU=1

# 是否在提示输入开头添加BOS Token：
# - 针对AdaptLLM模型，设为False。
# - 针对指令预训练模型，设为True。
# 若不确定，推荐设为False，该设置适用于绝大多数大语言模型。
add_bos_token=True

# 运行评估脚本
bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}

### 原始数据集
为便于微调或其他用途，我们还上传了原始训练与测试拆分数据集：ChemProt（https://huggingface.co/datasets/AdaptLLM/ChemProt）、RCT（https://huggingface.co/datasets/AdaptLLM/RCT）、ConvFinQA（https://huggingface.co/datasets/AdaptLLM/ConvFinQA）、FiQA_SA（https://huggingface.co/datasets/AdaptLLM/FiQA_SA）、Headline（https://huggingface.co/datasets/AdaptLLM/Headline）、NER（https://huggingface.co/datasets/AdaptLLM/NER）以及FPB（https://huggingface.co/datasets/AdaptLLM/FPB）。

### 领域知识探测
我们预处理后的知识探测数据集可在以下位置获取：生物医药知识探测数据集（med_knowledge_prob，https://huggingface.co/datasets/AdaptLLM/med_knowledge_prob）与法律知识探测数据集（law_knowledge_prob，https://huggingface.co/datasets/AdaptLLM/law_knowledge_prob）。

## 引用
若您认为我们的工作对您有所帮助，请引用以下文献：
bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}

以及原始数据集的引用：
bibtex
@inproceedings{MedMCQA,
author = {Ankit Pal and
Logesh Kumar Umapathi and
Malaikannan Sankarasubbu},
title = {MedMCQA: {A} Large-scale Multi-Subject Multi-Choice Dataset for Medical
domain Question Answering},
booktitle = {{CHIL}},
series = {Proceedings of Machine Learning Research},
volume = {174},
pages = {248--260},
publisher = {{PMLR}},
year = {2022}
}