Using cancer risk algorithms to improve risk estimates and referral decisions

Background: Cancer risk algorithms were introduced to clinical practice in the last decade, but they remain underused. We investigated whether GPs change their referral decisions in response to an unnamed algorithm, if decisions improve, and if changing decisions depends on having information about the algorithm and on whether GPs overestimated or underestimated risk.

Methods: 157 UK General Practitioners (GPs) were presented with 20 vignettes describing patients with possible colorectal cancer symptoms. GPs gave their risk estimates and inclination to refer. They then saw the risk score of an unnamed algorithm and could update their responses. Half of the sample was given information about the algorithm’s derivation, validation, and accuracy. At the end, we measured their algorithm disposition. We analysed the data using multilevel regressions with random intercepts by GP and vignette.

Results: We find that, after receiving the algorithm’s estimate, GPs’ inclination to refer changes 26% of the time and their decisions switch entirely 3% of the time. Decisions become more consistent with the NICE 3% referral threshold (OR 1.45 [1.27, 1.65], p<.001). The algorithm’s impact is greatest when GPs have underestimated risk. Information about the algorithm does not have a discernible effect on decisions but it results in a more positive GP disposition towards the algorithm. GPs’ risk estimates become better calibrated over time, i.e., move closer to the algorithm. 

Conclusions: Cancer risk algorithms have the potential to improve cancer referral decisions. Their use as learning tools to improve risk estimates is promising and should be further investigated. 

Methods
All data were collected online using the Qualtrics platform. The data were analysed using STATA 17.0 and R (version 4.0.3).