FiQA_SA

# Adapting Large Language Models to Domains via Continual Pre-Training
This repo contains the **FiQA_SA dataset** used in our **ICLR 2024** paper [Adapting Large Language Models via Reading Comprehension](https://huggingface.co/papers/2309.09530).

We explore **continued pre-training on domain-specific corpora** for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to **transform large-scale pre-training corpora into reading comprehension texts**, consistently improving prompting performance across tasks in biomedicine, finance, and law domains. **Our 7B model competes with much larger domain-specific models like BloombergGPT-50B**.

### [2024/11/29] 🤗 Introduce the multimodal version of AdaptLLM at [AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains), for adapting MLLMs to domains 🤗

**************************** **Updates** ****************************
* 2024/11/29: Released [AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains) for adapting MLLMs to domains
* 2024/9/20: Our [research paper for Instruction-Pretrain](https://huggingface.co/papers/2406.14491) has been accepted by EMNLP 2024
* 2024/8/29: Updated [guidelines](https://huggingface.co/datasets/AdaptLLM/finance-tasks) on evaluating any 🤗Huggingface models on the domain-specific tasks
* 2024/6/22: Released the [benchmarking code](https://github.com/microsoft/LMOps/tree/main/adaptllm)
* 2024/6/21: Released the general version of AdaptLLM at [Instruction-Pretrain](https://huggingface.co/instruction-pretrain)
* 2024/4/2: Released the [raw data splits (train and test)](https://huggingface.co/datasets/AdaptLLM/ConvFinQA) of all the evaluation datasets
* 2024/1/16: Our [research paper for AdaptLLM](https://huggingface.co/papers/2309.09530) has been accepted by ICLR 2024
* 2023/12/19: Released our [13B base models](https://huggingface.co/AdaptLLM/law-LLM-13B) developed from LLaMA-1-13B
* 2023/12/8: Released our [chat models](https://huggingface.co/AdaptLLM/law-chat) developed from LLaMA-2-Chat-7B
* 2023/9/18: Released our [paper](https://huggingface.co/papers/2309.09530), [code](https://github.com/microsoft/LMOps), [data](https://huggingface.co/datasets/AdaptLLM/law-tasks), and [base models](https://huggingface.co/AdaptLLM/law-LLM) developed from LLaMA-1-7B

## Domain-Specific LLaMA-1
### LLaMA-1-7B
In our paper, we develop three domain-specific models from LLaMA-1-7B, which are also available in Huggingface: [Biomedicine-LLM](https://huggingface.co/AdaptLLM/medicine-LLM), [Finance-LLM](https://huggingface.co/AdaptLLM/finance-LLM) and [Law-LLM](https://huggingface.co/AdaptLLM/law-LLM), the performances of our AdaptLLM compared to other domain-specific LLMs are:

<p align='center'>
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">
</p>

### LLaMA-1-13B
Moreover, we scale up our base model to LLaMA-1-13B to see if **our method is similarly effective for larger-scale models**, and the results are consistently positive too: [Biomedicine-LLM-13B](https://huggingface.co/AdaptLLM/medicine-LLM-13B), [Finance-LLM-13B](https://huggingface.co/AdaptLLM/finance-LLM-13B) and [Law-LLM-13B](https://huggingface.co/AdaptLLM/law-LLM-13B).

## Domain-Specific LLaMA-2-Chat
Our method is also effective for aligned models! LLaMA-2-Chat requires a [specific data format](https://huggingface.co/blog/llama2#how-to-prompt-llama-2), and our **reading comprehension can perfectly fit the data format** by transforming the reading comprehension into a multi-turn conversation. We have also open-sourced chat models in different domains: [Biomedicine-Chat](https://huggingface.co/AdaptLLM/medicine-chat), [Finance-Chat](https://huggingface.co/AdaptLLM/finance-chat) and [Law-Chat](https://huggingface.co/AdaptLLM/law-chat)

## Domain-Specific Tasks

### Pre-templatized/Formatted Testing Splits
To easily reproduce our prompting results, we have uploaded the filled-in zero/few-shot input instructions and output completions of the test each domain-specific task: [biomedicine-tasks](https://huggingface.co/datasets/AdaptLLM/medicine-tasks), [finance-tasks](https://huggingface.co/datasets/AdaptLLM/finance-tasks), and [law-tasks](https://huggingface.co/datasets/AdaptLLM/law-tasks).

**Note:** those filled-in instructions are specifically tailored for models before alignment and do NOT fit for the specific data format required for chat models.

### Raw Datasets
We have also uploaded the raw training and testing splits, for facilitating fine-tuning or other usages:
- [ChemProt](https://huggingface.co/datasets/AdaptLLM/ChemProt)
- [RCT](https://huggingface.co/datasets/AdaptLLM/RCT)
- [ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA)
- [FiQA_SA](https://huggingface.co/datasets/AdaptLLM/FiQA_SA)
- [Headline](https://huggingface.co/datasets/AdaptLLM/Headline)
- [NER](https://huggingface.co/datasets/AdaptLLM/NER)
- [FPB](https://huggingface.co/datasets/AdaptLLM/FPB)

The other datasets used in our paper have already been available in huggingface, and you can directly load them with the following code:
```python
from datasets import load_dataset

# MQP:
dataset = load_dataset('medical_questions_pairs')
# PubmedQA:
dataset = load_dataset('bigbio/pubmed_qa')
# USMLE:
dataset=load_dataset('GBaker/MedQA-USMLE-4-options')
# SCOTUS
dataset = load_dataset("lex_glue", 'scotus')
# CaseHOLD
dataset = load_dataset("lex_glue", 'case_hold')
# UNFAIR-ToS
dataset = load_dataset("lex_glue", 'unfair_tos')
```

## Citation
If you find our work helpful, please cite us:
```bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}
```

and the original dataset:
```bibtex
@inproceedings{FiQA-SA,
author = {Macedo Maia and
Siegfried Handschuh and
Andr{\'{e}} Freitas and
Brian Davis and
Ross McDermott and
Manel Zarrouk and
Alexandra Balahur},
title = {WWW'18 Open Challenge: Financial Opinion Mining and Question Answering},
booktitle = {{WWW} (Companion Volume)},
pages = {1941--1942},
publisher = {{ACM}},
year = {2018}
}
```

# 通过持续预训练适配大语言模型至特定领域
本仓库包含我们发表于**国际学习表征会议（ICLR）2024**的论文《通过阅读理解适配大语言模型》（Adapting Large Language Models via Reading Comprehension）[https://huggingface.co/papers/2309.09530] 中使用的**FiQA_SA数据集**。

我们探索了针对大语言模型（Large Language Model, LLM）的**领域专属语料库持续预训练**方案。尽管该方法可为大语言模型注入领域知识，但会显著损害其在问答任务中的提示学习能力。受人类通过阅读理解进行学习的启发，我们提出了一种简单方法，可将大规模预训练语料库**转换为阅读理解文本**，在生物医药、金融与法律领域的各类任务中持续提升提示学习性能。我们的70亿参数模型可与诸如BloombergGPT-50B这类超大规模领域专属模型相媲美。

### [2024/11/29] 🤗 我们在[AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains)推出了AdaptLLM的多模态版本，用于适配多模态大语言模型（Multimodal Large Language Model, MLLM）至特定领域 🤗

**************************** **更新日志** ****************************
* 2024/11/29：发布用于适配多模态大语言模型至特定领域的[AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains)
* 2024/9/20：我们关于指令预训练的研究论文[Instruction-Pretrain](https://huggingface.co/papers/2406.14491)已被自然语言处理经验方法会议（EMNLP）2024收录
* 2024/8/29：更新了用于在🤗Huggingface平台的领域专属任务上评估任意模型的[评估指南](https://huggingface.co/datasets/AdaptLLM/finance-tasks)
* 2024/6/22：发布了基准测试代码[benchmarking code](https://github.com/microsoft/LMOps/tree/main/adaptllm)
* 2024/6/21：在[Instruction-Pretrain](https://huggingface.co/instruction-pretrain)发布了AdaptLLM的通用版本
* 2024/4/2：发布了所有评估数据集的原始训练与测试划分数据[ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA)
* 2024/1/16：我们关于AdaptLLM的研究论文[AdaptLLM](https://huggingface.co/papers/2309.09530)已被ICLR 2024收录
* 2023/12/19：发布了基于LLaMA-1-13B开发的130亿参数基础模型[13B base models](https://huggingface.co/AdaptLLM/law-LLM-13B)
* 2023/12/8：发布了基于LLaMA-2-Chat-7B开发的聊天模型[chat models](https://huggingface.co/AdaptLLM/law-chat)
* 2023/9/18：发布了我们的论文[paper](https://huggingface.co/papers/2309.09530)、代码[code](https://github.com/microsoft/LMOps)、数据集[data](https://huggingface.co/datasets/AdaptLLM/law-tasks)以及基于LLaMA-1-7B开发的基础模型[base models](https://huggingface.co/AdaptLLM/law-LLM)

## 领域专属LLaMA-1
### LLaMA-1-7B
在我们的论文中，我们基于LLaMA-1-7B开发了三款领域专属模型，均已在Huggingface平台开源：[Biomedicine-LLM（生物医药大语言模型）](https://huggingface.co/AdaptLLM/medicine-LLM)、[Finance-LLM（金融大语言模型）](https://huggingface.co/AdaptLLM/finance-LLM)以及[Law-LLM（法律大语言模型）](https://huggingface.co/AdaptLLM/law-LLM)。我们的AdaptLLM与其他领域专属大语言模型的性能对比如下：

<p align='center'>
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">
</p>

### LLaMA-1-13B
此外，我们将基础模型扩展至LLaMA-1-13B规模，以验证**我们的方法在更大规模模型上同样有效**，实验结果同样呈现积极效果：对应的开源模型包括[Biomedicine-LLM-13B（生物医药大语言模型-13B）](https://huggingface.co/AdaptLLM/medicine-LLM-13B)、[Finance-LLM-13B（金融大语言模型-13B）](https://huggingface.co/AdaptLLM/finance-LLM-13B)以及[Law-LLM-13B（法律大语言模型-13B）](https://huggingface.co/AdaptLLM/law-LLM-13B)。

## 领域专属LLaMA-2-Chat
我们的方法对对齐后的模型同样有效！LLaMA-2-Chat需要特定的数据格式[specific data format](https://huggingface.co/blog/llama2#how-to-prompt-llama-2)，而我们的**阅读理解方法可将文本转换为多轮对话格式，完美适配该数据要求**。我们还开源了各领域的聊天模型：[Biomedicine-Chat（生物医药聊天模型）](https://huggingface.co/AdaptLLM/medicine-chat)、[Finance-Chat（金融聊天模型）](https://huggingface.co/AdaptLLM/finance-chat)以及[Law-Chat（法律聊天模型）](https://huggingface.co/AdaptLLM/law-chat)

## 领域专属任务
### 预模板化/格式化测试划分
为便于复现我们的提示学习实验结果，我们已上传各领域专属任务的填充完成的零样本/少样本输入指令与测试集输出结果：[biomedicine-tasks（生物医药任务数据集）](https://huggingface.co/datasets/AdaptLLM/medicine-tasks)、[finance-tasks（金融任务数据集）](https://huggingface.co/datasets/AdaptLLM/finance-tasks)以及[law-tasks（法律任务数据集）](https://huggingface.co/datasets/AdaptLLM/law-tasks)。

**注意**：上述填充完成的指令专为未经过对齐的模型设计，不适用于聊天模型所需的特定数据格式。

### 原始数据集
我们还上传了原始训练与测试划分数据集，以支持微调或其他应用场景：
- [ChemProt](https://huggingface.co/datasets/AdaptLLM/ChemProt)
- [RCT](https://huggingface.co/datasets/AdaptLLM/RCT)
- [ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA)
- [FiQA_SA](https://huggingface.co/datasets/AdaptLLM/FiQA_SA)
- [Headline](https://huggingface.co/datasets/AdaptLLM/Headline)
- [NER（命名实体识别，Named Entity Recognition）](https://huggingface.co/datasets/AdaptLLM/NER)
- [FPB](https://huggingface.co/datasets/AdaptLLM/FPB)

我们论文中使用的其他数据集已在Huggingface平台上线，可通过以下代码直接加载：
python
from datasets import load_dataset

# MQP:
dataset = load_dataset('medical_questions_pairs')
# PubmedQA（PubMed问答数据集）:
dataset = load_dataset('bigbio/pubmed_qa')
# USMLE（美国医学执照考试数据集）:
dataset=load_dataset('GBaker/MedQA-USMLE-4-options')
# SCOTUS（美国最高法院数据集）:
dataset = load_dataset("lex_glue", 'scotus')
# CaseHOLD:
dataset = load_dataset("lex_glue", 'case_hold')
# UNFAIR-ToS:
dataset = load_dataset("lex_glue", 'unfair_tos')

## 引用
如果您认为我们的工作对您有所帮助，请引用以下文献：
bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}

以及原始数据集的引用：
bibtex
@inproceedings{FiQA-SA,
author = {Macedo Maia and
Siegfried Handschuh and
Andr{e} Freitas and
Brian Davis and
Ross McDermott and
Manel Zarrouk and
Alexandra Balahur},
title = {WWW'18 Open Challenge: Financial Opinion Mining and Question Answering},
booktitle = {{WWW} (Companion Volume)},
pages = {1941--1942},
publisher = {{ACM}},
year = {2018}
}

搜集汇总

背景与挑战

背景概述

FiQA_SA是一个金融领域的情感分析数据集，用于提升大型语言模型在金融问答任务中的表现。该数据集源自WWW'18开放挑战赛，支持金融意见挖掘和问答研究。