Comparative Studies on Some Metrics for External Validation of QSPR Models

Quantitative structure–property relationship (QSPR)
models
used for prediction of property of untested chemicals can be utilized
for prioritization plan of synthesis and experimental testing of new
compounds. Validation of QSPR models plays a crucial role for judgment
of the reliability of predictions of such models. In the QSPR literature,
serious attention is now given to external validation for checking
reliability of QSPR models, and predictive quality is in the most
cases judged based on the quality of predictions of property of a
single test set as reflected in one or more external validation metrics.
Here, we have shown that a single QSPR model may show a variable degree
of prediction quality as reflected in some variants of external validation
metrics like Q2F1, Q2F2, Q2F3, CCC, and rm2 (all of which are
differently modified forms of predicted variance, which theoretically
may attain a maximum value of 1), depending on the test set composition
and test set size. Thus, this report questions the appropriateness
of the common practice of the “classic” approach of
external validation based on a single test set and thereby derives
a conclusion about predictive quality of a model on the basis of a
particular validation metric. The present work further demonstrates
that among the considered external validation metrics, rm2 shows statistically significantly different numerical
values from others among which CCC is the most optimistic or less
stringent. Furthermore, at a given level of threshold value of acceptance
for external validation metrics, rm2 provides
the most stringent criterion (especially with Δrm2 at highest tolerated value of 0.2) of external validation,
which may be adopted in the case of regulatory decision support processes.