CT-bias: a Dataset for Auditing the Impact of Patients' Sensitive Information on Clinical Trial Matching

Medical advances are rooted in insights gathered from clinical trials. Technology plays an increasing role in the clinical trial recruitment: patients can use specialised platforms to search for clinical trials, and clinical trials recruiters search through patient records to find eligible patients. The technology involved in medical search pipelines, such as matching patients to trials, is evolving rapidly. Existing research shows that personal characteristics of patients significantly influence the outputs of models tasked with medical question answering. While similar classes of neural models are increasingly present in medical information-seeking pipelines, the scientific community lacks resources to investigate how well these models cope with bias in medical retrieval tasks.In this paper, we present CT-bias, a novel resource for auditing of bias for specific patients and groups of patients where private or sensitive information was recoded as a part of a medical note. To create the dataset, we perturb patient notes from an established patient-trial matching benchmark by inserting specific patient information (sexual orientation, occupation, education level) irrelevant to their medical condition. In an example experiment we study the impact of this information on the retrieval results of 12 leading open source zero-shot dense retrieval models. We also propose a methodology to inspect models for biases. Our results show that, while most models are generally robust to medically irrelevant information, some patient characteristics impact the retrieval results more than others, thus showcasing the utility of our resource.