Estimation Stability with Cross Validation (ESCV)

Cross-validation (<i>CV</i>) is often used to select the regularization parameter in high dimensional problems. However, when applied to the sparse modeling method Lasso, <i>CV</i> leads to models that are unstable in high-dimensions, and consequently not suited for reliable interpretation. In this paper, we propose a model-free criterion <i>ESCV</i> based on a new <i>estimation stability (ES)</i> metric and <i>CV</i>. Our proposed <i>ESCV</i> finds a smaller and locally <i>ES</i>-optimal model smaller than the <i>CV</i> choice so that the it fits the data and also enjoys estimation stability property. We demonstrate that <i>ESCV</i> is an effective alternative to <i>CV</i> at a similar easily parallelizable computational cost. In particular, we compare the two approaches with respect to several performance measures when applied to the Lasso on both simulated and real data sets. For dependent predictors common in practice, our main finding is that, <i>ESCV</i> cuts down false positive rates often by a large margin, while sacrificing little of true positive rates. <i>ESCV</i> usually outperforms <i>CV</i> in terms of parameter estimation while giving similar performance as <i>CV</i> in terms of prediction. For the two real data sets from neuroscience and cell biology, the models found by <i>ESCV</i> are less than half of the model sizes by <i>CV</i>, but preserves <i>CV</i>'s predictive performance and corroborates with subject knowledge and independent work. We also discuss some regularization parameter alignment issues that come up in both approaches. Supplementary materials are available online.