Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation

BackgroundWith the large amount of biological data that is currently publicly available, many investigators combine multiple data sets to increase the sample size and potentially also the power of their analyses. However, technical differences (“batch effects”) as well as differences in sample composition between the data sets may significantly affect the ability to draw generalizable conclusions from such studies.FocusThe current study focuses on the construction of classifiers, and the use of cross-validation to estimate their performance. In particular, we investigate the impact of batch effects and differences in sample composition between batches on the accuracy of the classification performance estimate obtained via cross-validation. The focus on estimation bias is a main difference compared to previous studies, which have mostly focused on the predictive performance and how it relates to the presence of batch effects.DataWe work on simulated data sets. To have realistic intensity distributions, we use real gene expression data as the basis for our simulation. Random samples from this expression matrix are selected and assigned to group 1 (e.g., ‘control’) or group 2 (e.g., ‘treated’). We introduce batch effects and select some features to be differentially expressed between the two groups. We consider several scenarios for our study, most importantly different levels of confounding between groups and batch effects.MethodsWe focus on well-known classifiers: logistic regression, Support Vector Machines (SVM), k-nearest neighbors (kNN) and Random Forests (RF). Feature selection is performed with the Wilcoxon test or the lasso. Parameter tuning and feature selection, as well as the estimation of the prediction performance of each classifier, is performed within a nested cross-validation scheme. The estimated classification performance is then compared to what is obtained when applying the classifier to independent data.研究背景：当前公开可用的海量生物数据中，诸多研究者会整合多组数据集以扩大样本量，进而提升分析效力。但不同数据集间的技术差异（批次效应，batch effects）以及样本构成差异，可能会显著影响此类研究得出可推广结论的能力。 研究焦点：本研究聚焦分类器构建与交叉验证（cross-validation）在性能评估中的应用。具体而言，我们探究批次效应与批次间样本构成差异，对通过交叉验证得到的分类性能评估准确性的影响。相较于既往研究多聚焦预测性能及其与批次效应存在性的关联，本研究的核心差异在于对评估偏差的关注。 数据集构建：本研究采用模拟数据集。为获得贴合实际的表达强度分布，我们以真实基因表达数据（gene expression data）作为模拟基础：从该表达矩阵中随机抽取样本，并将其分配至组1（如“对照组”）或组2（如“处理组”）。我们引入批次效应，并筛选出部分在两组间存在差异表达的特征。本研究设置了多种实验场景，其中最关键的是组间混杂与批次效应的不同水平组合。 研究方法：本研究选用经典分类器：逻辑回归（logistic regression）、支持向量机（Support Vector Machines, SVM）、k近邻（k-nearest neighbors, kNN）与随机森林（Random Forests, RF）。特征筛选采用威尔科克森检验（Wilcoxon test）或套索回归（lasso）。参数调优、特征筛选以及各分类器的预测性能评估，均通过嵌套交叉验证（nested cross-validation）方案实现。最后，将得到的分类性能估计值，与分类器应用于独立数据集时的真实性能进行对比。