A Systematic Literature Review of Machine Learning for Uncovering Software Faults and Failures

This data set contains the results of an extensive, systematic literature review on the use of machine learning (ML) for uncovering software faults and failures. While existing literature reviews provide important insights into the use of ML for uncovering software faults and failures, they exhibit shortcomings with respect the following two concerns.

(1) Precise Terminology: Many terms are used in the software engineering literature -- often interchangeably -- to describe software quality problems, such as bug, error, defect, fault, and failure. Yet, to accurately classify the relevant papers, a precise definition of what software quality problems are addressed is needed. One precise distinction can be made between the cause of a software quality problem and the observation of a software quality problem during system execution.

The fundamental approaches followed by quality assurance tasks differ depending on whether they aim to observe a problem or whether they aim to understand and pinpoint the cause of a problem. Using such precise terminology provides a more fine-grained classification of the quality assurance tasks performed to uncover software quality problems than what existing literature reviews provide.  We clearly differentiate between the two kinds of software quality problems, calling the cause of a problem a fault and the observed problem a failure, which allows us in turn to classify the relevant papers into distinct types of quality assurance tasks.

Covering the period of 2019 to 2022, we identified 874 relevant papers in six distinct quality assurance tasks.

(2) Synergies among Quality Assurance Tasks: While existing literature reviews provide some high-level, relevant insights into research progress and remaining gaps in quality assurance, they treat the different quality assurance tasks in isolation, thus ignoring potential synergies that may exist between different quality assurance tasks. Yet, typically not a single type of quality assurance task or technique will suffice to deliver the expected software quality. Ideally, one would use a suitable set and combination of techniques, such as combining testing -- as a dynamic technique -- with static code analysis. In addition, when combining quality assurance techniques, one may increase the effectiveness of the individual techniques; e.g., having a good estimate of the fault density of a component may direct limited testing effort and budget.

Covering the period of 2019 to 2022, we identified 26 papers that explicitly leverage synergies, which is only 3% of all relevant papers, indicating ample opportunities for further exploring synergies in future research.