Replication Package for the Paper: "An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors".

This is the replication package for the paper: "An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors". The replication package is organized into three folders: 1. RQ1 Performance of LLMs - API invocation scriptThis folder contains the Python script used for calling LLMs via the API (run_llm.py), library versions required by the script (requirements.txt) and instructions for running it (readme.md). - Five prompt templates.pdfThis PDF demonstrates the detailed structures of the five prompt templates designed in Section 2.3.2 of our paper. - the cot-guardrail prompt template.pdfThis PDF presents the detailed structure of the sixth prompt template designed in Section 2.3.2 of our paper, which is the guardrail version of the CoT prompt. - source code of the Python and C/C++ datasetsThis folder contains the source code of the Python and C/C++ datasets, used to construct prompts and apply the baseline tools for static analysis. - prompts for the Python and C/C++ datasetsThis folder contains all prompts constructed from the source code of the Python and C/C++ datasets based on the six prompt templates. - responses of LLMs and baselinesThis folder contains the raw outputs generated by LLMs for each prompt and the analysis results of baseline tools. For CodeQL, you need to upload results.sarif to GitHub (https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github) to view the analysis results. For SonarQube, you need to import the export file into an Enterprise Edition or higher instance of the same version (v10.5 in our work) and similar configuration (default configuration in our work) to view the analysis results. - entropy_calculation.pyThis Python script calculates the average entropy of each llm-prompt combination to measure the consistency of LLM responses in three repetitive experiments. - Data Labelling for the C/C++ Dataset.xlsx- Data Labelling for the Python Dataset.xlsxThe two Microsoft (MS) files contain the labeling results for LLMs and baselines in the C/C++ and Python datasets, including the category of each response generated by LLM for each prompt, as well as the category of each analysis result generated by baseline for each code file. The four categories(i.e., Instrumental, Helpful, Misleading and Uncertain) are defined in Section 2.3.3 of our paper as the labelling criteria. How to Read the MS Excel files:Both MS Excel files contain 5 sheets. The first sheet ('all_c++_data' or 'all_python_data') includes the information of all data in each dataset. The sheets 'first round', 'second round' and 'third round' represent the labelling results for LLMs under five prompts in three repetitive experiments. The sheet 'Baselines' include the labelling results for baseline tools. Column Description File ID the identifier of each code file in our dataset. Security Defect the security defect(s) that the code file contains. Project the source project of the code file. Suffix the suffix of the code file. 2. RQ2 Quality Problem in Responses - DeepSeek-R1 data_analysis_first_round c.mx24 python.mx24 data_analysis_second_round c.mx24 python.mx24 data_analysis_third_round c.mx24 python.mx24 - GPT-4 data_analysis_first_round.mx24 data_analysis_second_round.mx24 data_analysis_third_round.mx24 This directory contains two sets of MAXQDA project files. They correspond to the data extraction results on quality problems in responses generated by the two best-performing LLMs: DeepSeek-R1 and GPT-4, under their optimal prompts across three repetitive experiments. We split the data extraction results of Deepseek-R1 in each round of experiments into two mx24 files by programming language (Python -> python.mx24, C/C++ -> c.mx24), to prevent a single file from being too large. The MAXQDA project file can be opened by MAXQDA 2024 or higher versions, which are available at https://www.maxqda.com/ for download. You may also use the free 14 days trial version of MAXQDA, which is available at https://www.maxqda.com/trial for download. 3. RQ3 Factor influencing LLMsThis folder contains two sub-folders: - Step 1 - correlation analysisThis subfolder contains the script and data for correlation analysis. phi_k_analysis.py : Python script for calculating correlations. DeepSeek-R1_factor.csv and GPT-4(ChatGPT)_factor.csv : Contain the values of all explanatory variables used to fit models for DeepSeek-R1 and GPT-4 (ChatGPT), respectively. - Step 2 - redundancy analysis and model fittingThis subfolder contains R scripts and data for cumulative link model fitting. simulation.R : Power analysis via Monte Carlo simulations. clm_regression.R : Redundancy analysis, allocation of degree of freedoms, model fitting and model evaluation. readme.md : Detailed instructions for running the R scripts. DeepSeek-R1.csv and GPT-4(ChatGPT).csv: Contain the values of all explanatory and response variables, used to fit models when running clm_regression.R. Note that the versions of R packages required by the scripts are specified in comments within the scripts.