Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Prediction Errors for Deep Neural Networks

Deep
learning architectures have proved versatile in a number of
drug discovery applications, including the modeling of in
vitro compound activity. While controlling for prediction
confidence is essential to increase the trust, interpretability, and
usefulness of virtual screening models in drug discovery, techniques
to estimate the reliability of the predictions generated with deep
learning networks remain largely underexplored. Here, we present Deep
Confidence, a framework to compute valid and efficient confidence
intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically,
Deep Confidence generates an ensemble of deep neural networks by recording
the network parameters throughout the local minima visited during
the optimization phase of a single neural network. This approach serves
to derive a set of base learners (i.e., snapshots) with comparable
predictive power on average that will however generate slightly different
predictions for a given instance. The variability across base learners
and the validation residuals are in turn harnessed to compute confidence
intervals using the conformal prediction framework. Using a set of
24 diverse IC50 data sets from ChEMBL 23, we show that
Snapshot Ensembles perform on par with Random Forest
(RF) and ensembles of independently trained deep neural networks.
In addition, we find that the confidence regions predicted using the
Deep Confidence framework span a narrower set of values. Overall,
Deep Confidence represents a highly versatile error prediction framework
that can be applied to any deep learning-based application at no extra
computational cost.