Can Program Analysis Tools Find More Code Review Defects?

Program analysis tools find defects in code, checking code against rules to reveal potential defects. Many studies have evaluated these tools by measuring their ability to detect known defects in code. But these studies measure the current state of tools rather than their future potential to find more defects. To investigate the prospects for tools to find more defects, we conducted a study where we examined the potential to create rules, which might be checked by tools, which would prevent defects found in code review. We first gathered a corpus of 1323 defects found through code review. Through a qualitative analysis process, for each defect we identified a violated rule and the type of program analysis tool (PAT) which might check this rule.We found that PATs might, in principle, be used to detect as many as 76% of code review defects, considerably more than current tools have been demonstrated to successfully detect. Among a variety of types of program analysis tools, Style Checkers and AST Pattern Checkers had the broadest coverage of defects, each with the potential to detect 25% of all code review defects. To detectable defects, existing tools require features like supporting custom and more complex rules. Defects not detectable by PATs require human judgement and are violations of rules lacking formalism.