Data set for ``Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?''

One of the black arts of data mining is learning the magic parameters that control the learners. In software analytics, at least for defect prediction, several methods, like grid search and differential evolution(DE), have been proposed to learn those parameters. They've been proved to be able to improve learner performance.

We want to evaluate which method can find better parameters in terms of performance score and runtime. This paper compares grid search to differential evolution, which is an evolutionary algorithm that makes extensive use of stochastic jumps around the search space. We find that the seemingly complete approach of grid search does no better, and sometimes worse, than the stochastic search. Yet, when repeated 20 times to check for conclusion validity, DE was over 210 times faster (6.2 hours for DE vs 54 days for grid search when both tuning Random Forest over 17 test data sets with F-measure as optimization objective).

These results are puzzling: why does a quick partial search be just as effective as a much slower, and much more, extensive search? To answer that question, we turned to the theoretical optimization literature. Bergstra and Bengio conjecture that grid search is not more effective than more randomized searchers if the underlying search space is inherently low dimensional. This is significant since recent results show that defect prediction exhibits very low intrinsic dimensionality– an observation that explains why a fast method like DE may work as well as a seemingly more thorough grid search. This suggests, as a future research direction, that it might be possible to peek at data sets before doing any optimization in order to match the optimization algorithm to the problem at hand.