nsk7153/MedCalc-Bench

--- license: cc-by-sa-4.0 dataset_info: features: - name: Row Number dtype: int64 - name: Calculator ID dtype: int64 - name: Calculator Name dtype: string - name: Category dtype: string - name: Output Type dtype: string - name: Note ID dtype: string - name: Note Type dtype: string - name: Patient Note dtype: string - name: Question dtype: string - name: Relevant Entities dtype: string - name: Ground Truth Answer dtype: string - name: Lower Limit dtype: string - name: Upper Limit dtype: string - name: Ground Truth Explanation dtype: string splits: - name: train num_bytes: 41265307 num_examples: 10053 - name: test num_bytes: 4038342 num_examples: 1047 download_size: 19626866 dataset_size: 45303649 configs: - config_name: default data_files: - split: train path: data/train-* - split: test path: data/test-* --- MedCalc-Bench is the first medical calculation dataset used to benchmark LLMs ability to serve as clinical calculators. Each instance in the dataset consists of a patient note, a question asking to compute a specific clinical value, an final answer value, and a step-by-step solution explaining how the final answer was obtained. Our dataset covers 55 different calculation tasks. We hope this dataset serves as a call to improve the verbal and computational reasoning skills of LLMs in medical settings. This dataset contains a training dataset of 10.1k instances and a testing dataset of 1047 instances. ## Contents inside the Training and Testing CSV In total, there are 1047 instances. Each row in the dataset contains the following information Calculator Name, Note ID, Note Type (Extracted (if note is from Open-Patients), Synthetic (if note is handwritten by clinician), or Template (if note is generated using a template)), Relevant Entities needed for calculation, Ground Truth Answer, Lower Limit, Upper Limit, Ground Truth Explanation. Note that for equation-based calculators whose output is a decimal, the values for the Upper and Lower Limit will be +/- 0.05% of the ground truth answer. If the LLM's final answer value is between the upper and lower limit, the answer is considered correct. For all other instances, the Upper Limit and Lower Limit are set to the same value as the ground truth. We make error accomodation for equation-based LLM calculators to accomodate for any rounding differences in intermediate steps. This issue is nonexistant for date-based equation calculators and rule-based calculators where final value is independent any rounding done in the intermediate steps. ## How to Use MedCalc-Bench The training dataset of MedCalc-Bench can be used for fine-tunining LLMs. We have provided both the fine-tuned models and code for fine-tuning at our repository link: https://github.com/ncbi-nlp/MedCalc-Bench. The test set of MedCalc-Bench is helpful for benchamrking LLMs under different settings. We provide instructions in the README of our repository for how to reproduce all of our results for all of the models using the different prompt settings. By experimenting with different LLMs and prompts, we hope our dataset demonstrates the potential and limitations of LLMs in different settings. ## License Both the training and testing dataset of MedCalc-Bench are released under the CC-BY-SA 4.0 license.

原始信息汇总

数据集概述

数据集信息

• 特征列表：

  • Row Number: 数据类型为 int64
  • Calculator ID: 数据类型为 int64
  • Calculator Name: 数据类型为 string
  • Category: 数据类型为 string
  • Output Type: 数据类型为 string
  • Note ID: 数据类型为 string
  • Note Type: 数据类型为 string
  • Patient Note: 数据类型为 string
  • Question: 数据类型为 string
  • Relevant Entities: 数据类型为 string
  • Ground Truth Answer: 数据类型为 string
  • Lower Limit: 数据类型为 string
  • Upper Limit: 数据类型为 string
  • Ground Truth Explanation: 数据类型为 string

• 数据分割：

  • train: 包含 10053 个样本，大小为 41265307 字节
  • test: 包含 1047 个样本，大小为 4038342 字节

• 数据集大小：

  • 下载大小：19626866 字节
  • 数据集总大小：45303649 字节

数据集配置

• 默认配置：
  • train 数据文件路径：data/train-*
  • test 数据文件路径：data/test-*

数据集内容

• 训练和测试数据集：

  • 训练集包含 10100 个实例
  • 测试集包含 1047 个实例
  • 每个实例包含以下信息：
    • Calculator Name
    • Note ID
    • Note Type（可能为 Extracted、Synthetic 或 Template）
    • Relevant Entities
    • Ground Truth Answer
    • Lower Limit
    • Upper Limit
    • Ground Truth Explanation

• 特殊说明：

  • 对于输出为小数的基于方程的计算器，上下限为真实答案的 +/- 0.05%
  • 对于其他实例，上下限与真实答案相同

数据集用途

• 训练数据集：用于微调大型语言模型（LLMs）
• 测试数据集：用于在不同设置下基准测试大型语言模型（LLMs）

许可证

• 数据集发布在 CC-BY-SA 4.0 许可证下