Information Sharing for Robust and Stable Cross-Validation

Robust estimators for linear regression require non-convex objective functions to shield against adverse effects of contamination, including outliers. This non-convexity brings challenges, particularly when combined with penalization in high-dimensional settings. A crucial challenge is selecting hyperparameters for the penalty based on a finite sample. In practice, cross-validation (CV) is the prevalent strategy with good performance for convex estimators. Applied with robust estimators, however, CV often gives subpar results due to the interplay between multiple local minima and the penalty. The best local minimum attained on the full training data may not be the minimum with the desired statistical properties. Furthermore, there may be a mismatch between this minimum and the minima attained in the CV folds which are used for evaluating the prediction error. This article introduces a novel adaptive CV strategy that tracks multiple minima for each combination of hyperparameters and subsets of the data. A matching scheme is presented for correctly evaluating minima computed on the full training data using the best-matching minima from the CV folds. We show that the proposed strategy reduces the variability of the estimated performance metric, leads to smoother CV curves, and therefore substantially increases the reliability and utility of robust penalized estimators. Supplementary materials for this article are available online.