CWE and Severity Identification Artifacts

Triaging security vulnerabilities is a critical part of software maintenance, ensuring that the most pressing issues are addressed promptly to safeguard system integrity and user data before any catastrophe occurs. However, this process is resource-intensive, time-consuming, and presents several challenges, including classifying software vulnerabilities, assessing their severity, and managing a high volume of bug reports. In this experience paper, we explore the use of a Large Language Model (LLM)—specifically, the GPT model—to determine whether it can automate CWE identification and severity assessment of security vulnerabilities. Our pipeline, CASEY, leverages prompt engineering and incorporates contextual information at varying levels of code granularity. We evaluated CASEY on an augmented dataset derived from the National Vulnerability Database (NVD), using both quantitative and qualitative analyses. CASEY achieves a CWE identification accuracy of 68%, a severity identification accuracy of 73.6%, and a combined accuracy of 51.2%, which reflects the strengths and limitations of LLM-based automation for identification and assessment of security vulnerabilities. Our findings demonstrate the potential of LLMs in automating vulnerability triage. We also discuss the challenges encountered, insights gained during development, and offer practical advice for researchers and practitioners aiming to deploy LLMs in real-world vulnerability identification and assessment workflows.