LLM-SAT-Eval

Exploring the theme “Do static analysis tools foresee actual bug fixes?," this poster investigates the practical utility of Static Analysis Tools (SATs) in anticipating genuine bug fixes within software code-bases. Leveraging a dataset comprising bug-fix pairs, our study scrutinizes the efficacy of SATs through quantitative analysis, revealing a significant disparity between SAT outputs and actual bug fixes, with nearly 93% of pairs rated at 0. Employing qualitative insights from Gemini-pro Large Language Model, we corroborate these findings, observing prevalent terms such as "buggy line" and "not able" in bug detection. Furthermore, correlation analysis showcases a moderate positive correlation (0.3) between SAT outputs and known bug fixes, emphasizing the importance of alignment for favorable ratings. Our findings underscore the pressing need for enhanced bug detection methodologies, prompting further research in software quality assurance.