Assessing and Improving Prompting Large Language Models for Software Vulnerability Analysis (TOSEM'26)

# Assessing and Improving Prompting Large Language Models for Software Vulnerability Analysiswe explore how to leverage LLMs and CoT to address three key software vulnerability analysis tasks: identifying a given type of vulnerabilities, discovering vulnerabilities of any type, and patching detected vulnerabilities. We instantiate the general CoT methodology in the context of these tasks through vulnerability semantics-guided prompting (VSP) which is our unified vulnerability-semantics-guided prompting approach, and conduct extensive experiments assessing VSP versus five baselines for the three tasks against three LLMs and three datasets. ## Package Structure- `VSP_artifact.zip`: The exemplars, testing datasets, and results for VSP prompting and the baselines. - `datasets`: The testing datasets used for evaluating LLMs and the baselines. - `vsp_dataset`: The real-world dataset based on the BigVul+CVEFixes. - `vulchecker_dataset_func`: The real-world dataset based on the VulChecker work. - `zeroday_dataset`: The zeroday dataset where all the samples are after the LLMs' cutoff dates. - `detection`: The exemplars and the results for task1 -- vulnerability detection. - `exemplars`: The exemplars used for few-shot learning for LLMs. - `exemplars_lbl`: The exemplars for the baseline `Naive CoT Learning`. - `exemplars_vsp`: The exemplars for our proposed `VSP prompting`. - `LLMs`: The results of LLMs on the three datasets. - `baselines`: The results of traditional vulnerability analysis techniques. - `classification`: The exemplars and the results for task1 -- vulnerability classification. - `exemplars`: The exemplars used for few-shot learning for LLMs. - `exemplars_lbl`: The exemplars for the baseline `Naive CoT Learning`. - `exemplars_vsp`: The exemplars for our proposed `VSP prompting`. - `LLMs`: The results of LLMs on the three datasets. - `baselines`: The results of traditional vulnerability analysis techniques. - `patching`: The exemplars and the results for task3 -- vulnerability patching. - `exemplars`: The exemplars used for few-shot learning for LLMs. - `exemplars_lbl`: The exemplars for the baseline `Naive CoT Learning`. - `exemplars_vsp`: The exemplars for our proposed `VSP prompting`. - `vsp_repair_dataset`: The dataset and results of LLMs and baselines on our collected dataset based on BigVul+CVEFixes. - `extractfix`: The dataset and results of LLMs and baselines on our collected dataset based on ExtractFix. - `zeroday_repair`: The dataset and results of LLMs and baselines on our collected dataset based on the Zeroday dataset. ## How to use Please use the package structure to find the exemplars, evaluation data, and results for the corresponding contents described in the original paper.