Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

Vulnerable Code Generation with Open-weight LLMs Overview This repository contains the scripts and setup to evaluate the generation of vulnerable code by open-weight Large Language Models (LLMs) in response to various prompts. We use dynamic prompting (zero-shot) and reverse prompting (one-shot) techniques to guide the models (Qwen2, Mistral, and Gemma) to generate C code that contains specific vulnerabilities. Accepted Paper https://arxiv.org/abs/2507.10054: @misc{bosnak2025explicitvulnerabilitygenerationllms, title={Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks}, author={Emir Bosnak and Sahand Moslemi and Mayasah Lami and Anil Koyuncu}, year={2025}, eprint={2507.10054}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10054}, } Repository Structure . ├── analysis - Contains analysis and vulnerability detection scripts │ ├── esbmc - ESBMC analysis tools and scripts │ ├── esbmc-reports - Raw ESBMC output logs │ ├── full_analysis_dynamic_prompting.py - Dynamic prompting analysis script │ ├── full_analysis_reverse_prompting.py - Reverse prompting analysis script │ └── vuln-analysis-summary - ESBMC analysis summaries ├── c-code - Extracted C source files from model outputs │ ├── dynamic - C files from dynamic prompting │ └── reverse - C files from reverse prompting ├── compact_horizontal_plot_with_backgrounds.png - Figure visualization ├── final-summary - Final analysis reports │ ├── dynamic_prompting_full_analysis.txt - Dynamic prompting results │ └── reverse_prompting_full_analysis.txt - Reverse prompting results ├── FormAI-v2 - FormAI-v2 dataset files │ └── FormAI-v2.json - Metadata and vulnerability annotations ├── full_generation_visual.py - Figure 5 generation script ├── heatmap_correct_vuln.png - Heatmap visualization ├── heatmap.py - Figure 6 generation script ├── kaggle-notebooks - Jupyter notebooks for model inference │ ├── gemma.ipynb - Gemma model notebook │ ├── gemma_temp.ipynb - Gemma temperature experiment │ ├── mistral.ipynb - Mistral model notebook │ ├── mistral_temp.ipynb - Mistral temperature experiment │ ├── qwen2.ipynb - Qwen2 model notebook │ └── qwen2_temp.ipynb - Qwen2 temperature experiment ├── model-outputs - JSON files containing model generation results │ ├── dynamic - Dynamic prompting model outputs │ └── reverse - Reverse prompting model outputs ├── prompt-generation - Prompt generation scripts and files │ ├── dynamic - Dynamic prompting generation │ └── reverse - Reverse prompting generation ├── README.md - This file ├── requirements.txt - Python dependencies └── sampled_code - Sampled vulnerable code by vulnerability type ├── buffer-overflow - Buffer overflow samples ├── dereference-failure - Dereference failure samples ├── integer-overflow - Integer overflow samples ├── out-of-bounds - Out-of-bounds access samples └── use-after-free - Use-after-free samples Requirements OS and Environment The experiments were run on Kaggle Notebooks using default settings. The analysis scripts were run on Ubuntu 24.04 TLS Python 3.x environment with the following dependencies: transformers accelerate einops bitsandbytes These dependencies are listed in the requirements.txt file. Models Qwen2-7B-Instruct Mistral-7B-Instruct Gemma-7B-Instruct These models were loaded using the Hugging Face transformers library with quantization options for efficient inference. GPU/Hardware The experiment was conducted on Kaggle using their GPU P100 accelerator. Additional Software ESBMC (Extended Symbolic Model Checker) - latest version from GitHub 7zip for dataset extraction Gemini API key (for reverse prompt generation) You can apply the reviewer's comment by merging the redundant content between step 5 of Installation and the Dataset Setup section. Here's how to reorganize it: Installation Install the necessary Python dependencies: pip install -r requirements.txt Install 7zip: sudo apt install 7zip Install ESBMC: Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo (github.com/esbmc/esbmc): # Download the latest esbmc-linux.zip from ESBMC GitHub Releases unzip esbmc-linux.zip -d esbmc echo 'export PATH="$HOME/esbmc/bin:$PATH"' >> ~/.bashrc source ~/.bashrc Verify ESBMC installation: esbmc --version Expected output: ESBMC version information Basic usage test - Generate dynamic prompts: sudo python3 prompt-generation/dynamic/dynamic_prompt_generation.py Expected output: Saved 2040 prompts to /prompt-generation/dynamic/dynamic_prompts.json Test ESBMC integration by extracting sample code: python analysis/esbmc/extract_code.py --input path_to_model_output_file This will create compilable C files in the ./c-code directory, confirming that the code extraction and compilation validation pipeline is working correctly. Installation tested on: Ubuntu 24.04 TLS with Python 3.x, using Kaggle GPU P100 accelerator for model inference. Dataset Setup FormAI-v2 Dataset This project builds on the FormAI-v2 dataset, which provides vulnerable C code samples generated by large language models. To set up the FormAI-v2 dataset: Clone the repository: git clone https://github.com/FormAI-Dataset/FormAI-dataset/ Extract the classification files: 7z x FormAI-dataset/FormAI-v2-classification.7z.001 This command will automatically handle both parts (.7z.001 and .7z.002) and extract the FormAI-v2.json file. Move the dataset to the correct location: mkdir -p FormAI-v2 mv FormAI-v2.json FormAI-v2/ Dataset Structure FormAI-v2.json contains metadata and vulnerability annotations for each file in the dataset. Each entry includes: filename: name of the C source file (also the model name in the file name) vuln_line and vuln_column: the location of the vulnerability in the code violation: the violated ESBMC property (i.e., the vulnerability type) snippet: a short code excerpt showing the vulnerable region cyclomatic_complexity: a complexity score used for bucketed analysis Experiment Commands Dynamic Prompting (Zero-shot) - prompt-generation/dynamic Generate the Prompts: To generate the dynamic prompts, run the following script: sudo python3 prompt-generation/dynamic/dynamic_prompt_generation.py Since sudo uses Python 2 by default, using sudo python ... might result in an error. Expected stdout: Saved 2040 prompts to /prompt-generation/dynamic/dynamic_prompts.json Run the Models on Prompts - ./kaggle-notebooks The models are run using the respective Jupyter notebooks for Qwen2, Mistral, and Gemma, located in the ./kaggle-notebooks directory. The generated prompts are provided as input to these notebooks by setting the variable prompt_path = "/kaggle/input/new-prompts/dynamic_prompts.json". The outputs will be saved to results/<model_name>_results.json. ./model_outputs contains these JSON files. Sample 100 Prompts (For Temperature Experiment) If you wish to run the temperature experiment with a sample of prompts, run: python prompt-generation/dynamic/sample_prompts.py Expected stdout: Saved 100 prompts to .../prompt-generation/dynamic/dynamic_prompts_temp_sample.json Reverse Prompting (One-Shot) - sampled_code Generation - prompt-generation/reverse Extract Vulnerable Code from FormAI-v2: python prompt-generation/reverse/extract_vulnerable_code.py Expected stdout: [done] Extracted 742252 vulnerable files into .../prompt-generation/reverse/formai_all_vulnerable_code The script takes input from ./FormAI-v2/FormAI-v2.json. It filters valid, verified vulnerable code samples, categorizes them by vulnerability type and cyclomatic complexity, and saves the source code as .c files in ./prompt-generation/reverse/formai_all_vulnerable_code/<vuln_type>/<cc_bucket>/. Get a Sample of Vulnerable Code: python prompt-generation/reverse/get_sample_code.py The script takes input from ./prompt-generation/reverse/formai_all_vulnerable_code/, randomly samples up to 5 .c files per subtype and cyclomatic complexity bucket, and saves them to ./prompt-generation/reverse/sampled_code/<vuln_type>/<subtype>/<cc_bucket>/. It organizes samples by vulnerability type, subtype, and CC bucket, resolving subtype-to-folder mappings where needed. Generate Reverse Prompts Using Gemini API (or any other LLM): Add your Gemini API key to the generate_reverse_prompts.py file and run: python prompt-generation/reverse/generate_reverse_prompts.py You can also change the Gemini setup and call_llm(prompt) function with a commercial LLm of your choice. However, this could impact the final results. ESBMC Setup - /analysis Install ESBMC Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo (github.com/esbmc/esbmc) and add the binary to your environment's path. 1.1 Download: Get the latest esbmc-linux.zip from the ESBMC GitHub Releases. 1.2 Extract: unzip esbmc-linux.zip -d esbmc 1.3. Add esbmc/bin to PATH: echo 'export PATH="$HOME/esbmc/bin:$PATH"' >> ~/.bashrc source ~/.bashrc 1.4. Verify: esbmc --version Run ESBMC Analysis Run ESBMC Analysis After collecting model outputs, extract C code with: python analysis/esbmc/extract_code.py --input path_to_model_output_file Enter the model result filename (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json): Before running the script, ensure that the model outputs are placed in the following path format: ./model-outputs/<dynamic|reverse>/<Model>/<JSON File> (e.g., ./model-outputs/reverse/qwen2/qwen2_reverse_prompting_full.json). Then, provide the relative path from ./model-outputs as the input to process the file (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json). The output will be stored in the ./c-code directory. The script saves the extracted .c source files and temporary .out binaries generated during compilation. Files are saved under the following structure: ../../c-code/<mode>/<model>/ Example (based on user input reverse/qwen2/qwen2_reverse_prompting_full.json): ../../c-code/reverse/qwen2/ After attempting to compile each .c file, the script removes: Any .c files that fail to compile All temporary .out binary files generated during compilation Only compilable .c files are retained in the output directory. Run ESBMC on Extracted Code python analysis/esbmc/esbmc_analysis.py --input path_to_extracted_code Before running this script, ensure that you have already generated the extracted .c files by completing the previous step. This script analyzes the extracted C files using ESBMC. It expects the same relative model result path provided in the previous step (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json) as input. It performs the following actions: Locates the corresponding .c files under ./c-code/<dynamic|reverse>/<Model>/, based on the input path. Runs ESBMC on each .c file with a 15-second timeout, using verification flags such as --incremental-bmc and --overflow-check. Writes raw ESBMC output logs to ./analysis/esbmc-reports/<mode>/, one .txt file per C file. Generates two JSON summaries in ./analysis/vuln-analysis-summary/<mode>/: A detailed summary with ESBMC results, matched metadata (e.g., vulnerability type, prompt), and violated properties if any. An executive summary with counts of each result type and, if applicable, a breakdown by temperature. Timeouts, parsing errors, and verification failures are identified and reported. Only .c files in the extracted code directory are processed. Full Analysis Once the C code is extracted and analyzed, run the following scripts for full analysis: cd analysis python full_analysis_dynamic_prompting.py python full_analysis_reverse_prompting.py These will generate detailed analysis summaries, which are stored in the ./final-summary directory. Figures & Tables full_generation_visual.py and heatmap.py use hardcoded data from ./final-summary and generate figures 5 and 6 in the paper respectively. The results in tables 1 to 4 are manually taken from ./final-summary. ./final-summary includes two files, ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt. Each file is organized by model and research question. The RQ1: Vulnerable‐code and correct‐vuln counts part in ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt contains results for Table-1 The RQ2: By intent (absolute counts) parts contains results for Table-2. The RQ3: By form (absolute counts) parts contains results for Table-3. The RQ4: Incorrect Vulnerability Generation Patterns parts contains results for Table-4. Notes The "total_prompts": 2040, "code_generated": 2034, "compilable_code": 1711 attributes in the executive summary need to be added manually by checking the results from ./analysis/esbmc/extract_code.py (this part was not automated). For reverse prompt generation, Gemini API key is required. You can use any other LLM API, but be aware that results may differ significantly due to different model architectures and training data. Outputs and Results All intermediate and final results—including model generations, C code files, ESBMC reports, and summaries—are included in the replication release v1.0. These outputs are required to reproduce the tables and findings in the paper and are not tracked in the main branch due to their size. Final Remarks This repository provides a comprehensive framework for evaluating the security risks associated with the use of LLMs in generating code. The techniques can be adapted to other tasks involving code generation, vulnerability analysis, and prompt engineering for LLMs.