Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection

This is the replication package accompanying our paper "Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection." Codebase structure The project is structured as follows. .├── scripts/ # bash scripts to run Gumtree├── src/ # source code of the project├── raw_predictions # the original predictions (patches + interpretations) made by 5 LLMs ├── processed_data # the processed data├── requirements.txt # required Python libraries Run SusVF Step 1. Run Gumtree bash scripts/gumtree.sh processed_data/src processed_data/<pred_pack> where pred_pack is the directory containing processed data of predictions made by an LLM under a prompting technique (zero-shot CoT or few-shot CoT) Step 2. Filter Gumtree's output python gumtree_filter.py \ --diff_dir processed_data/<pred_pack>/gumtree_diff \--ast_dir processed_data/<pred_pack>/gumtree_ast \--src_dir processed_data/src \--dst_dir processed_data/<pred_pack>/src \--out_dir processed_data/<pred_pack>/gumtree_filter Step 3. Generate NL patch descriptions python gpt_gumtree2nl.py \ --diff_dir processed_data/<pred_pack>/gumtree_filter \--patch_desc_dir processed_data/<pred_pack>/patch_desc \ Step 4. Run SusVF's main program python susvf_main.py \ --data_file processed_data/<pred_pack>/merge_data.csv \--working_dir=processed_data/<pred_pack> \--src_dir=processed_data/src