disi-unibo-nlp/medmcqa-MedGENIE
收藏数据集卡片 "medmcqa-MedGENIE"
数据集描述
该数据集是MedGENIE系列医学数据集的一部分，通过PMC-LLaMA-13B生成的合成上下文进行了增强。具体来说，每个MedMCQA中的问题最多生成了5个合成上下文，采用多视角方法来涵盖与给定问题相关的各种视角。
该数据集已被用于训练MedGENIE-fid-flan-t5-base-medmcqa，使其在MedMCQA和MMLU-Medical基准测试中达到与最先进（SOTA）大型模型相媲美的性能水平。
更多信息请参考我们的论文"To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering"。
数据集结构
该数据集包含两个拆分，适用于：
- 训练问答模型，包括融合-解码器架构。
- 在推理过程中使用生成的上下文增强大型语言模型（LLMs），而不是检索的块。
- 使用生成的上下文增强事实文档的知识库，用于标准RAG管道。
每个拆分的样本数量为：
- train: 182822个样本
- validation: 4183个样本
数据集以parquet格式存储，每个条目使用以下模式： json { "id_question": "45258d3d-b974-44dd-a161-c3fccbdadd88", "question": "Which of the following is not true for myelinated nerve fibers: A. Impulse through myelinated fibers is slower than non-myelinated fibers B. Membrane currents are generated at nodes of Ranvier C. Saltatory conduction of impulses is seen D. Local anesthesia is effective only when the nerve is not covered by myelin sheath", "target": "A", "answers": [ "A" ], "ctxs": [ { "text": "The myelin sheath of myelinated nerve fibers is a covering that acts as insulation and increases the rate of conduction. Therefore, impulse through myelinated fibers is faster than non-myelinated fibers. Understanding these differences in structure and function between these two types of nerve cells helps us appreciate..." }, { "text": "The myelin sheath is a protective layer that wraps around the nerve fibers and helps to speed up the transmission of signals along nerves. Each fiber has its own insulating cover, known as a Schwann cell membrane, which forms the intermediate substance (or endoneurium) between the perineurium and epineurium sheaths..." }, { "text": "This question focuses on the differences between myelinated and non-myelinated nerve fibers. Myelinated fibers are wrapped in a sheath that helps them transmit action potentials more rapidly. The impulse jumps from one node to the next at a rate of 2-3 meters/second. In contrast, non-myelinated fibers do not have this..." }, { "text": "Myelinated nerve fibers are those that have a sheath of myelin surrounding them. In the central nervous system, they are called tracts and can be up to 4 inches (10 cm) in length; in contrast, the peripheral nervous system has much smaller fiber bundles called a fascicle. The myelin is produced by Schwann cells which..." }, { "text": "Myelinated nerve fibersu00a0are those that have a fatty, protective sheath called a myelin coating, which is formed from the membranes of Schwann cells. Some sensory and motor fibers in the peripheral nervous system are unmyelinated (i.e., they lack a myelin covering). Because they conduct impulses more rapidly than..." } ] }
增强LLMs在推理过程中的应用
使用medmcqa-MedGENIE和medqa-MedGENIE生成的上下文增强最先进的LLMs，显示了显著的性能提升。对于给定的问题，所有相关的上下文都被连接并传递到LLM的上下文窗口中。
| 模型 | 学习方式 | medqa-5-opt-MedGENIE | 准确率 |
|---|---|---|---|
| LLaMA-2-chat (7B) | 2-shot | NO | 35.0 |
| LLaMA-2-chat (7B) | 2-shot | YES | 44.1 (+ 9.1) |
| Zephyr-β (7B) | 2-shot | NO | 43.4 |
| Zephyr-β (7B) | 2-shot | YES | 50.5 (+ 7.1) |
RAG评估
为了评估使用我们生成的上下文进行RAG管道的有效性，我们使用从medmcqa-MedGENIE中派生的较小部分人工生成的块来增强MedWiki数据集。
| MedWiki块 | 人工块 | 重新排序 | LLaMA-2-chat (7B) | mistral-instruct (7B) | Zephyr-β (7B) |
|---|---|---|---|---|---|
| 4.5M | - | NO | 35.8 | 44.2 | 47.0 |
| 4.5M | 96K | NO | 40.9 (+ 5.1) | 47.4 (+ 3.2) | 50.1 (+3.1) |
| 4.5M | 2M | NO | 43.1 (+ 7.3) | 48.6 (+ 4.4) | 50.9 (+3.9) |
| 4.5M | - | YES | 37.3 | 44.3 | 47.1 |
| 4.5M | 96K | YES | 41.8 (+4.5) | 48.1 (+3.8) | 50.3 (+3.2) |
| 4.5M | 2M | YES | 43.7 (+6.4) | 49.8 (+5.5) | 50.7 (+3.6) |
引用
如果您发现此数据集在您的工作中有用，请引用：
@misc{frisoni2024generate, title={To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering}, author={Giacomo Frisoni and Alessio Cocchieri and Alex Presepi and Gianluca Moro and Zaiqiao Meng}, year={2024}, eprint={2403.01924}, archivePrefix={arXiv}, primaryClass={cs.CL} }
