Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management

We introduce Q-Pain, a dataset for assessing bias in medical QA in the context of pain management. We developed 55 medical question-answer pairs across five different types of pain management: each question includes a detailed patient-specific medical scenario ("vignette") designed to enable the substitution of multiple different racial and gender "profiles" and to evaluate whether bias is present when answering whether or not to prescribe medication.

Along with the dataset, we propose a new framework, including a sample experimental design, to measure the potential bias present during medical decision-making for patients with particular demographic profiles. We demonstrate its use in assessing two reference QA systems, GPT-2 and GPT-3, selected for their documented ability to answer questions given only a few examples. We hope that our dataset and framework will be used to assess bias across a wide variety of pain management QA systems.

本团队推出Q-Pain数据集，旨在评估在疼痛管理背景下医学问答中的偏见。该数据集包含涵盖五种不同类型疼痛管理的55个医学问答对；每个问题均包含详细的针对特定患者的医疗场景（“情景剧”），旨在实现多种不同种族和性别“档案”的替换，并评估在回答是否开具药物时是否存在偏见。伴随数据集，我们提出了一种新的框架，包括一个样本实验设计，用于测量在针对特定人口统计特征的患者进行医学决策过程中可能存在的偏见。我们展示了该框架在评估两个参考问答系统GPT-2和GPT-3中的应用，这两者因其仅在少量示例的基础上回答问题的记录能力而被选中。我们期望我们的数据集和框架能够被广泛应用于评估各类疼痛管理问答系统的偏见。