fairness_bias.pdf

Objective: The study aims to investigate whether machine learning-based predictive models for cardiovascular

disease (CVD) risk assessment show equivalent performance across demographic groups (such as race and

gender) and if bias mitigation methods can reduce any bias present in the models. This is important as systematic

bias may be introduced when collecting and preprocessing health data, which could affect the performance of the

models on certain demographic sub-cohorts. The study is to investigate this using electronic health records data

and various machine learning models.

Methods: The study used large de-identified Electronic Health Records data from Vanderbilt University Medical

Center. Machine learning (ML) algorithms including logistic regression, random forest, gradient-boosting trees,

and long short-term memory were applied to build multiple predictive models. Model bias and fairness were

evaluated using equal opportunity difference (EOD, 0 indicates fairness) and disparate impact (DI, 1 indicates

fairness). In our study, we also evaluated the fairness of a non-ML baseline model, the American Heart Association

(AHA) Pooled Cohort Risk Equations (PCEs). Moreover, we compared the performance of three different

de-biasing methods: removing protected attributes (e.g., race and gender), resampling the imbalanced training

dataset by sample size, and resampling by the proportion of people with CVD outcomes.

Results: The study cohort included 109,490 individuals (mean [SD] age 47.4 [14.7] years; 64.5% female; 86.3%

White; 13.7% Black). The experimental results suggested that most ML models had smaller EOD and DI than

PCEs. For ML models, the mean EOD ranged from 􀀀 0.001 to 0.018 and the mean DI ranged from 1.037 to 1.094

across race groups. There was a larger EOD and DI across gender groups, with EOD ranging from 0.131 to 0.136

and DI ranging from 1.535 to 1.587. For debiasing methods, removing protected attributes didn’t significantly

reduced the bias for most ML models. Resampling by sample size also didn’t consistently decrease bias.

Resampling by case proportion reduced the EOD and DI for gender groups but slightly reduced accuracy in many

cases.

Conclusions: Among the VUMC cohort, both PCEs and ML models were biased against women, suggesting the

need to investigate and correct gender disparities