Replication Package for the Paper: "A Multi-Agent Framework with Dynamic Context Querying in Security Defect Detection".

This is the replication package for the paper:  "A Multi-Agent Framework with Dynamic Context Querying in Security Defect Detection".

The replication package is organized into two folders:

Folder 1. Chapter3-empirical_study

1.1 RQ1 Performance of LLMs

- API invocation scriptThis folder contains the Python script used for calling LLMs via the API (run_llm.py), library versions required by the script (requirements.txt) and instructions for running it (readme.md).

- Five prompt templates.pdfThis PDF demonstrates the detailed structures of the five prompt templates designed in Section 3.1.3.2 of our paper.

-  the cot-guardrail prompt template.pdfThis PDF presents the detailed structure of the sixth prompt template designed in Section 3.1.3.2 of our paper, which is the guardrail version of the CoT prompt.

- source code of the Python and C/C++ datasetsThis folder contains the source code of the Python and C/C++ datasets, used to construct prompts and apply the baseline tools for static analysis.

- prompts for the Python and C/C++ datasetsThis folder contains the prompts constructed from the source code of the Python and C/C++ datasets based on the six prompt templates.

- responses of LLMs and baselinesThis folder contains the responses generated by LLMs for each prompt and the analysis results of baseline tools. For CodeQL, you need to upload results.sarif to GitHub (https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github) to view the analysis results. For SonarQube, you need to import the export file into an Enterprise Edition or higher instance of the same version (v10.5 in our work) and similar configuration (default configuration in our work) to view the analysis results. 

- entropy_calculation.pyThis Python script calculates the average entropy of each llm-prompt combination to measure the consistency of LLM responses in three repetitive experiments.

- Data Labelling for the C/C++ Dataset.xlsx- Data Labelling for the Python Dataset.xlsxThe two Microsoft (MS) files contain the labeling results for LLMs and baselines in the C/C++ and Python datasets, including the category of each response generated by LLM for each prompt, as well as the category of each analysis result generated by baseline for each code file.  The four categories(i.e., Instrumental, Helpful, Misleading and Uncertain) are defined in Section 3.1.3.3 of our paper as the labelling criteria.

How to Read the MS Excel files:Both MS Excel files contain 5 sheets. The first sheet ('all_c++_data' or 'all_python_data') includes the information of all data in each dataset. The sheets 'first round', 'second round' and 'third round' represent the labelling results for LLMs under five prompts in three repetitive experiments. The sheet 'Baselines' include the labelling results for baseline tools.

Column
Description

File ID
the identifier of each code file in our dataset.

Security Defect
the security defect(s) that the code file contains.

Project  
the source project of the code file.

Suffix
the suffix of the code file.

1.2 RQ2 Quality Problem in Responses

- DeepSeek-R1

data_analysis_first_round

c.mx24

python.mx24

data_analysis_second_round

c.mx24

python.mx24

data_analysis_third_round

c.mx24

python.mx24

- GPT-4(ChatGPT)

data_analysis_first_round.mx24

data_analysis_second_round.mx24

data_analysis_third_round.mx24

This directory contains two sets of MAXQDA project files. They correspond to the data extraction results on quality problems in responses generated by the two best-performing LLMs: DeepSeek-R1 and GPT-4, under their optimal prompts across three repetitive experiments. We split the data extraction results of Deepseek-R1 in each round of experiments into two mx24 files by programming language (Python -> python.mx24, C/C++ -> c.mx24), to prevent a single file from being too large. The MAXQDA project file can be opened by MAXQDA 2024 or higher versions, which are available at https://www.maxqda.com/ for download. You may also use the free 14 days trial version of MAXQDA, which is available at https://www.maxqda.com/trial for download.

1.3 RQ3 Factor influencing LLMsThis folder contains two sub-folders:

- Step 1 - correlation analysisThis subfolder contains the script and data for correlation analysis.

phi_k_analysis.py : Python script for calculating correlations.

DeepSeek-R1_factor.csv and GPT-4(ChatGPT)_factor.csv : Contain the values of all explanatory variables used to fit models for DeepSeek-R1 and GPT-4 (ChatGPT), respectively.

- Step 2 - redundancy analysis and model fittingThis subfolder contains R scripts and data for cumulative link model fitting.

simulation.R : Power analysis via Monte Carlo simulations.

clm_regression.R : Redundancy analysis, allocation of degree of freedoms, model fitting and model evaluation.

readme.md : Detailed instructions for running the R scripts.

DeepSeek-R1.csv and GPT-4(ChatGPT).csv: Contain the values of all explanatory and response variables, used to fit models when running clm_regression.R.

Note that the versions of R packages required by the scripts are specified in comments within the scripts.

Folder 2. Chapter4-multi-agent_framework

2.1 RQ1

- source outputThis folder contains the source outputs of all agent-based approaches under the evaluation dataset, including DynCPG-LLM, CoT, GPTLens and VulTrial. 

- evaluation resultThis folder contains the evaluation results of each source output in RQ1 based on the evaluation model. Specifically, each output is compared against the ground truth and categorized as either a 'match' or a 'mismatch' (or 'false alarm'). This process aims to measure the performance of different methods. For further details, please refer to Section 4.2.3.2.

- RQ1_evaluation_result.xlsxThis MS Excel file lists detailed evaluation results for each output in RQ1 to facilitate a visual representation of performance.

- evaluated_benchmark.jsonlThis JSONL file is the evaluation dataset utilized in RQ1 for DynCPG-LLM and CoT.

- evaluated_benchmark_vultrai_gptlens.jsonlThis JSONL file is the evaluation dataset utilized in RQ1 for GPTLens and VulTrial. While its data content is identical to that of 'evaluated_benchmark.jsonl', its structure has been modified—specifically, paired vulnerability samples have been split into individual samples—to meet the code requirements of the replication packages for GPTLens and VulTrial.

2.2 RQ2

- source outputThis folder contains the source outputs for different architectures in the ablation experiments, including Rev, Rev+Retr, Rev+Crit, and Rev+Retr+Crit, conducted under the evaluation dataset.

- evaluation resultThis folder contains the evaluation results of each source output in RQ2 based on the evaluation model.

- RQ2_evaluation_result.xlsxThis MS Excel file lists detailed evaluation results for each output in RQ2.

- evaluated_benchmark.jsonl and evaluated_benchmark_vultrai_gptlens.jsonlIdentical to those in RQ1.

2.3 RQ3

- security_code_review_c_dataset.jsonlThis JSONL file is the C/C++ dataset utilized in Chapter 3, which consists of security code reviews with security defects. The dataset is different from the evaluation dataset used in RQ1 of Chapter 4, thus adopted to measure the generalization of  DynCPG-LLM.

- source outputThis folder contains the source outputs of all agent-based approaches under the C/C++ dataset, including DynCPG-LLM, P_cot−guardrail, GPTLens and VulTrial. 

- RQ3_evaluation_result.xlsxThis MS Excel file lists detailed evaluation results for each output in RQ3.

2.4 manual_check_evaluation

- manual_check_result.xlsxIn Section 4.2.3.2,  we randomly selected 50 vulnerability sample pairs and manually labeled whether the approach detected the target security defects to validate the reliability of evaluation model. This .xlsx file records the results of the data labelling.

Column
Description

Sample_id

Unique identifier for the vulnerability sample pair.

Code

Code diff from the fix commit corresponding to the vulnerability sample.

CVE
CWE-ID and detailed CVE description of the vulnerability.

Vulnerable_detection
Vulnerability detection result produced by the method on the vulnerable version of the code.

Vulnerable_evaluation
The output of evaluation model for Vulnerable_detection.

Vulnerable_manual_check
Manual data labelling for Vulnerable_detection.

Non_vulnerable_detection
Vulnerability detection result produced by the method on the non-vulnerable version of the code.

Non_vulnerable_evaluation
The output of evaluation model for Non_vulnerable_detection.

Non_vulnerable_manual_check
Manual data labelling for Non_vulnerable_detection.

2.5 model_selection

- source outputThis folder contains the source outputs generated by DynCPG-LLM on the evaluation dataset when using different base models (i.e., Qwen2.5-Coder32B-Instruct and Qwen3-Coder-30B-A3B-Instruct) for the CPGQL query generation model.

- evaluation resultThis folder contains the evaluation results of each source output during model selection based on the evaluation model.

-  evaluation_result_of_qwen3_qwen2.5.xlsxThis MS Excel file lists detailed evaluation results for each output in model selection.