Effectiveness Assessment of Open-Source Static Analysis Tools

This spreadsheet presents data regarding the effectiveness of four open-source static analysis tools (SATs), namely PMD, SpotBugs, SonarLint, and Infer. These tools were evaluated considering a set of 35 Java projects collected from well-known benchmarks, such as Defects4J, QuixBugs, and NpeFix Dataset. The data set shows the number of true/false positives/negatives identified from the analysis performed by each SAT. Assessing the number of false positives/negatives is particularly essential for analyzing SATs since they might report warnings about faults that do not exist or miss warnings about actual faults. While false negatives impact the reliability of SATs because faults are missed, false positives lead developers to waste time on investigating incorrect warnings in the program under analysis, thus having a greater potential of negatively affecting the usability of SATs. These measures of true/false positives/negatives can be used to determine both precision and recall of each SAT, which are relevant metrics in this context as SATs are known for generating false positives/negatives, which may limit their adoption.

For the effectiveness assessment, a manual analysis was carried out upon the execution output of each tool for each sample. In the 134 executions finished without tool-related errors, a process of classification and partitioning was carried out to filter out only indications of NPE-related faults, which are the object of interest of this work. False negatives were considered as the amount of known NPE-related faults that were not detected in a sample. This amount was obtained by considering the number of known faults informed by the selected benchmarks or the ones identified by manually inspecting the small-sized samples.