Ensemble of Classifiers for Cross-Project Bug Prediction A Replicated Study

Bug prediction models are used to locate source code elements more likely to be defective. Broadly speaking, such models rely on a set of independent variables to predict the bug-proneness of source code elements using a machine learner. One of the key factor influencing their performances is related to the selection of a machine learning method (a.k.a., classifier) to use when discriminating buggy and non-buggy classes: indeed, previous research showed that a wrong selection might lead to a performance decrease up to 30%. More importantly, different classifiers tend to perform similarly even tough they are able to correctly predict the bug-proneness of different code elements. As a consequence, the research community focused on methods suitable to combine different classifiers, i.e., the so-called ensemble techniques. A variety of approaches have been proposed and to understand which of them are more reliable in the context of cross-project bug prediction some empirical investigations have been performed. As a result, Liu et al. and Zhang et al. found that the Validation and Voting technique is the one performing better with respect to the others. However, in the context of our research we figured out four important limitations of previous empirical studies aimed at benchmarking ensemble methods that possibly led to unreliable results: (i) the datasets exploited in previous work have been shown to be noisy, (ii) some data preprocessing techniques, whose application is highly recommended, have not been considered by previous analyses, (iii) the evaluation schema adopted relied on threshold-dependent metrics that do not provide a clear understanding of the results, and (iv) the size of previous studies threats the generalizability of the observed findings. To cope with these issues, in this paper we provide a replicated study that considers a cleaned dataset composed of 21 systems, where we analyze the behavior of seven ensemble methods. The results show that the problem is still far from being solved and the use of ensemble techniques does not provide evident benefits with respect to stand-alone classifiers. Finally, it confirms, in the context of ensemble-based models, the findings of previous studies that demonstrated that cross-project bug prediction models perform worse than within-project ones, being however more robust to performance variability.