Fractional cross-validation for optimizing hyperparameters of supervised learning algorithms

K-fold cross-validation (CV) is a robust method for estimating generalization performance of supervised learning models. Although CV is more reliable than using a single hold-out test set, it is also more computationally expensive since the model must be fit K times. This can be prohibitive when optimizing the hyperparameters, since this involves conducting K-fold CV repeatedly at many hyperparameter configurations. In this work, we propose a highly-efficient Bayesian optimization algorithm for optimizing the hyperparameters of supervised learning algorithms with K-fold CV error as the evaluation criterion. Our approach exploits the fact that the single-fold out-of-sample error is pairwise correlated across different hyperparameter configurations. We introduce a hierarchical Gaussian process model that is well-suited to accommodate this inherent correlation structure across folds and across the hyperparameter space. Our resulting algorithm requires evaluating only a single fold for many hyperparameter configurations, enabling us to efficiently find the optimal hyperparameters. We refer to this as “fractional CV”, since it requires only a small fraction of the folds to be evaluated, relative to what is required for full K-fold CV. We demonstrate the efficacy of our method on a number of models and real datasets.

K折交叉验证（K-fold cross-validation, CV）是评估监督学习模型泛化性能的稳健方法。尽管K折交叉验证比单一留出测试集更可靠，但计算成本也更高——因为模型需要被训练K次。在优化超参数的场景中，这一成本往往难以承受，因为需要在大量超参数配置下重复执行K折交叉验证。本研究提出了一种高效的贝叶斯优化算法，以K折交叉验证误差作为评估准则，用于优化监督学习算法的超参数。我们利用了不同超参数配置下的单折样本外误差存在成对相关性这一特性，并引入了分层高斯过程模型，可很好地适配折间与超参数空间中的固有相关结构。所提算法仅需在大量超参数配置下评估单折误差，从而能够高效寻得最优超参数。我们将该方法称为"部分折交叉验证（fractional CV）"，因为相较于完整K折交叉验证的需求，它仅需评估极小一部分折次即可完成计算。我们在多种模型与真实数据集上验证了所提方法的有效性。