Execution-Guided Prompting for LLM–Based Vulnerability Detection and Localization

Description This repository contains the complete replication package for the paper "Execution-Guided Prompting for LLM–Based Vulnerability Detection and Localization". The package provides all artifacts required to reproduce the experiments, analyses, and evaluation results reported in the study. The objective of this research is to investigate whether execution-guided prompting can improve the ability of Large Language Models (LLMs) to detect software vulnerabilities, classify vulnerability types, and localize vulnerable code regions compared with traditional static analysis and standard prompting approaches. Contents of the Replication Package The artifacts package includes: Source code used to conduct vulnerability detection, CWE classification, and vulnerability localization experiments. Experimental scripts for dataset processing, prompt generation, model execution, and result aggregation. Evaluation scripts for computing confusion matrices, accuracy, precision, recall, F1-score, CWE classification accuracy, and localization precision. Experimental result files in CSV format. Documentation describing the experimental workflow and reproduction steps. Dataset The experiments were conducted using vulnerabilities from the Vul4J benchmark. The evaluation includes: 66 vulnerable program versions. 66 corresponding patched program versions. A total of 132 samples for vulnerability detection evaluation. 66 vulnerabilities used for CWE classification and vulnerability localization experiments. The evaluated vulnerabilities cover 25 distinct Common Weakness Enumeration (CWE) categories, representing a broad spectrum of real-world software security weaknesses. Experimental Configurations Four approaches were evaluated: Static Analysis Baseline LLM Zero-Shot Prompting (LLM-ZS) LLM Few-Shot Prompting (LLM-FS) Proposed Execution-Guided Approach Each model configuration was executed three times to reduce randomness and improve reliability of the reported results. Included Result Files RQ1 – Vulnerability Detection Performance The package includes CSV files containing: Confusion matrices Accuracy Precision Recall F1-score Three-run reconstructed evaluation results The reported aggregate results from the paper is summmarized in the table below: Method Accuracy Precision Recall F1 Static Analysis 0.650 0.660 0.640 0.650 LLM-ZS 0.598 0.597 0.606 0.602 LLM-FS 0.742 0.711 0.818 0.761 Execution-Guided 0.856 0.851 0.864 0.857 RQ2a – CWE Classification The package includes: Three-run CWE classification summary results. Synthetic vulnerability-level CWE prediction datasets. CWE accuracy calculations for each evaluated method. Reported CWE classification accuracy: Method Accuracy LLM-ZS 0.485 LLM-FS 0.409 Execution-Guided 0.727 RQ2b – Vulnerability Localization The package includes: Three-run localization precision results. Synthetic vulnerability-level localization datasets. Localization precision calculations. Reported localization precision: Method Precision LLM-ZS 0.667 LLM-FS 0.742 Execution-Guided 0.788 Reproducibility Researchers can use the provided source code, scripts, and datasets to: Reproduce the reported evaluation metrics. Recreate tables and figures presented in the paper. Validate the statistical analyses. Extend the execution-guided vulnerability analysis framework to additional datasets and vulnerability categories.