Replication Package: Evaluating LLM-Derived Features in Software Defect Prediction: A Comparison with Conventional Approach

Software Defect Prediction (SDP) is a well-researched domain with numerous innovative solutions proposed to enhance the defect prediction performance from software systems. Despite these advancements, the introduction of software bugs or defects remains a common challenge, impacting the acceptability and effectiveness of any software system. Transformer-based Large Language Models (LLMs) have revolutionized various fields of Software Engineering, including program comprehension, code generation, and information retrieval. In this study, we investigate the use of popular Transformer-based LLMs, such as Generative Pre-trained Transformer (GPT) and three variants of Bidirectional Encoder Representations from Transformer (BERT, codeBERT, and graphCodeBERT) to convert commit patches of two popular software projects (OpenStack and QT) into feature vectors for predicting software defects using seven popular Machine Learning (ML) and Deep Learning (DL) models. We compare the performance of these LLM-Derived Features (LDF) with Conventional Features (CF) used in SDP. Our investigation compares various scenarios, such as comparing the defect prediction performance among the LDFs, LDF vs CF, and combining each LDF and CF. Our results reveal that while LLM-derived features (LDF) demonstrate strong predictive capabilities, conventional features (CF) also perform comparably well in several cases, underscoring their continued relevance in SDP. Our findings demonstrate the potential of using LDF, CF, and their combination to improve the accuracy and reliability of SDP, paving the way for more robust and defect-free software systems.