Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection

This is the replication package accompanying our paper "Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection."

Codebase structure

The project is structured as follows.

.├── scripts/                        # bash scripts to run Gumtree├── src/                              # source code of the project├── raw_predictions         # the original predictions (patches + interpretations) made by 5 LLMs

├── processed_data         # the processed data├── requirements.txt        # required Python libraries

Run SusVF

Step 1. Run Gumtree 

bash scripts/gumtree.sh processed_data/src processed_data/<pred_pack>

where pred_pack is the directory containing processed data of predictions made by an LLM under a prompting technique (zero-shot CoT or few-shot CoT)

Step 2. Filter Gumtree's output

python gumtree_filter.py \

--diff_dir processed_data/<pred_pack>/gumtree_diff \--ast_dir  processed_data/<pred_pack>/gumtree_ast \--src_dir  processed_data/src \--dst_dir  processed_data/<pred_pack>/src \--out_dir  processed_data/<pred_pack>/gumtree_filter

Step 3. Generate NL patch descriptions

python gpt_gumtree2nl.py \

--diff_dir processed_data/<pred_pack>/gumtree_filter \--patch_desc_dir processed_data/<pred_pack>/patch_desc \

Step 4. Run SusVF's main program

python susvf_main.py \

--data_file processed_data/<pred_pack>/merge_data.csv \--working_dir=processed_data/<pred_pack> \--src_dir=processed_data/src