Modelling methods and cross-validation variants in QSAR: a multi-level analysis^$

Prediction performance often depends on the cross- and test validation protocols applied. Several combinations of different cross-validation variants and model-building techniques were used to reveal their complexity. Two case studies (acute toxicity data) were examined, applying five-fold cross-validation (with random, contiguous and Venetian blind forms) and leave-one-out cross-validation (CV). External test sets showed the effects and differences between the validation protocols. The models were generated with multiple linear regression (MLR), principal component regression (PCR), partial least squares (PLS) regression, artificial neural networks (ANN) and support vector machines (SVM). The comparisons were made by the sum of ranking differences (SRD) and factorial analysis of variance (ANOVA). The largest bias and variance could be assigned to the MLR method and contiguous block cross-validation. SRD can provide a unique and unambiguous ranking of methods and CV variants. Venetian blind cross-validation is a promising tool. The generated models were also compared based on their basic performance parameters (<i>r</i>² and <i>Q</i>²). MLR produced the largest gap, while PCR gave the smallest. Although PCR is the best validated and balanced technique, SVM always outperformed the other methods, when experimental values were the benchmark. Variable selection was advantageous, and the modelling had a larger influence than CV variants.

预测性能往往取决于所采用的交叉验证与测试验证协议。为揭示此类验证流程的复杂性，本研究采用了多种不同交叉验证变体与模型构建技术的组合方案。本研究选取两组急性毒性（acute toxicity）数据集作为案例，分别采用五折交叉验证（含随机、连续块及百叶窗式（Venetian blind）三种划分形式）与留一交叉验证（CV）。外部测试集用于验证不同验证协议间的效果与差异。模型构建采用了多元线性回归（multiple linear regression, MLR）、主成分回归（principal component regression, PCR）、偏最小二乘回归（partial least squares regression, PLS）、人工神经网络（artificial neural networks, ANN）以及支持向量机（support vector machines, SVM）五种方法。研究采用排序差异之和（sum of ranking differences, SRD）与析因方差分析（factorial analysis of variance, ANOVA）开展对比分析。偏差与方差最大的情况分别出现在MLR方法与连续块交叉验证中。SRD可对各类建模方法与交叉验证变体实现唯一且明确的排序，百叶窗式交叉验证是一种颇具应用前景的验证工具。本研究还基于模型的基础性能参数（<i>r</i>²与<i>Q</i>²）对生成的模型进行了对比：MLR生成的模型性能差异跨度最大，而PCR生成的模型性能差异跨度最小。尽管PCR是验证最为充分且平衡性最佳的建模技术，但当以实验测量值作为基准时，SVM始终优于其他所有方法。此外，变量选择对建模具有积极作用，且建模方法对结果的影响程度大于交叉验证变体。