Towards Automatic Labeling of Exception Handling Bugs: A Case Study of 10 Years Bug-Fixing in Apache Hadoop

<strong>Context:</strong> Exception handling (EH) bugs stem from incorrect usage of exception handling mechanisms (EHMs) and often incur severe consequences (e.g., system downtime, data loss, and security risk). Tracking EH bugs is particularly relevant for contemporary systems (e.g., cloud- and AI-based systems), in which the software's sophisticated logic is an additional threat to the correct use of the EHM. On top of that, bug reporters seldom can tag EH bugs --- since it may require an encompassing knowledge of the software's EH strategy.
Surprisingly, to the best of our knowledge, there is no automated procedure to identify EH bugs from report descriptions.
<br>
<strong>Objective:</strong> First, we aim at evaluating the extent to which Natural Language Processing (NLP) and Machine Learning (ML) can be used to reliably label EH bugs using the text fields from bug reports (e.g., summary, description, and comments). Second, we aim at providing a reliably labeled dataset that the community can use  in future endeavors. Overall, we expect our work to raise the community's awareness regarding the importance of EH bugs.
<br>
<strong>Method:</strong> We manually analyzed 4,516 bug reports from the four main components of Apache’s Hadoop project, out of which we labeled  ~ 943 as EH bugs. Then, we used word embedding techniques (Bag-of-Words and TF-IDF) to summarize the textual fields of bug reports. Subsequently, we used these embeddings to fit four classes of ML methods and record their performance on unseen data. We have also evaluated whether considering only EH keywords is enough to achieve high predictive performance.
<br>
<strong>Results:</strong> Our results show that the combination of NLP and ML techniques can label EH bugs reasonably well, achieving ROC-AUC scores of up to 0.70 and recall ranging from 0.50 up to 0.62. As a sanity check, we also evaluate methods using embeddings extracted solely from keywords. While keyword-based embeddings yield similar AUC, we observe a steep decrease in recall (0.53). This suggests that keywords alone are not sufficient to characterize reports of EH bugs --- and there is an avenue for more complex text analyses. 
<br>
<strong>Conclusions:</strong> To the best of our knowledge, this is the first study addressing the problem of automatic labeling of EH bugs. Based on our results, we can conclude that the combination of NLP and ML techniques sounds promising to automate the task of labeling EH bugs.   Overall, we hope (i) that our work will contribute towards raising awareness around EH bugs; and (ii) that our (publicly available) dataset will serve as a benchmarking dataset, paving the way for follow-up works. Additionally, our findings can be used to build tools that help maintainers flesh out EH bugs during the triage process.