Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

Vulnerable Code Generation with Open-weight LLMs Overview This repository contains the scripts and setup to evaluate the generation of vulnerable code by open-weight Large Language Models (LLMs) in response to various prompts. We use dynamic prompting (zero-shot) and reverse prompting (one-shot) techniques to guide the models (Qwen2, Mistral, and Gemma) to generate C code that contains specific vulnerabilities. Accepted Paper https://arxiv.org/abs/2507.10054: @misc{bosnak2025explicitvulnerabilitygenerationllms, title={Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks}, author={Emir Bosnak and Sahand Moslemi and Mayasah Lami and Anil Koyuncu}, year={2025}, eprint={2507.10054}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.10054}, } Repository Structure Summary . ├── analysis - Contains analysis and vulnerability detection scripts │ ├── esbmc - ESBMC analysis tools and scripts │ ├── esbmc-reports - Raw ESBMC output logs │ ├── full_analysis_dynamic_prompting.py - Dynamic prompting analysis script │ ├── full_analysis_reverse_prompting.py - Reverse prompting analysis script │ └── vuln-analysis-summary - ESBMC analysis summaries ├── c-code - Extracted C source files from model outputs │ ├── dynamic - C files from dynamic prompting │ └── reverse - C files from reverse prompting ├── compact_horizontal_plot_with_backgrounds.png - Figure visualization ├── final-summary - Final analysis reports │ ├── dynamic_prompting_full_analysis.txt - Dynamic prompting results │ └── reverse_prompting_full_analysis.txt - Reverse prompting results ├── FormAI-v2 - FormAI-v2 dataset files │ └── FormAI-v2.json - Metadata and vulnerability annotations ├── full_generation_visual.py - Figure 5 generation script ├── heatmap_correct_vuln.png - Heatmap visualization ├── heatmap.py - Figure 6 generation script ├── kaggle-notebooks - Jupyter notebooks for model inference │ ├── gemma.ipynb - Gemma model notebook │ ├── gemma_temp.ipynb - Gemma temperature experiment │ ├── mistral.ipynb - Mistral model notebook │ ├── mistral_temp.ipynb - Mistral temperature experiment │ ├── qwen2.ipynb - Qwen2 model notebook │ └── qwen2_temp.ipynb - Qwen2 temperature experiment ├── model-outputs - JSON files containing model generation results │ ├── dynamic - Dynamic prompting model outputs │ └── reverse - Reverse prompting model outputs ├── prompt-generation - Prompt generation scripts and files │ ├── dynamic - Dynamic prompting generation │ └── reverse - Reverse prompting generation ├── README.md - This file ├── requirements.txt - Python dependencies └── sampled_code - Sampled vulnerable code by vulnerability type ├── buffer-overflow - Buffer overflow samples ├── dereference-failure - Dereference failure samples ├── integer-overflow - Integer overflow samples ├── out-of-bounds - Out-of-bounds access samples └── use-after-free - Use-after-free samples Requirements Summary OS and Environment The experiments were run on Kaggle Notebooks using default settings. The analysis scripts were run on Ubuntu 24.04 TLS Python ≥ 3.11 environment with the following dependencies: transformers accelerate einops bitsandbytes These dependencies are listed in the requirements.txt file. Models Qwen2-7B-Instruct Mistral-7B-Instruct Gemma-7B-Instruct These models were loaded using the Hugging Face transformers library with quantization options for efficient inference. GPU/Hardware The experiment was conducted on Kaggle using their GPU P100 accelerator. For NVIDIA driver issues, see: https://documentation.ubuntu.com/server/how-to/graphics/install-nvidia-drivers/index.html Additional Software ESBMC (Extended Symbolic Model Checker) - latest version from GitHub: https://github.com/esbmc/esbmc 7zip for dataset extraction Gemini API key (for reverse prompt generation) (https://aistudio.google.com/app/apikey)[https://aistudio.google.com/app/apikey] Hugging Face Access Token with the "Read access to contents of all public gated repos you can access" permission (https://huggingface.co/settings/tokens)[https://huggingface.co/settings/tokens] Installation Notes Some dependencies may require additional system packages not explicitly listed Detailed instructions to install ESBMC and 7zip are provided in Installation section NVIDIA drivers and CUDA packages are required to run notebooks in the kaggle-notebook directory, which can be run using Kaggle https://kaggle.com. The rest of the components can be run without CUDA packages and NVIDIA drivers. Installation Steps 1. Install the necessary Python dependencies: pip install -r requirements.txt Note: Python versions older than 3.11 may encounter compatibility errors with some dependencies. 2. Install 7zip: sudo apt install 7zip Note: If you encounter NVIDIA driver compilation errors during package installation (like DKMS build failures), this is unrelated to 7zip installation. The errors occur because broken or incompletely installed NVIDIA drivers prevent other packages from configuring properly. Example error you might see: dpkg: error processing package nvidia-dkms-575 (--configure): Building module(s)..............(bad exit status: 2) Error! Bad return status for module build on kernel To verify 7zip installed correctly: 7z --help To fix NVIDIA driver issues: Complete the 7zip installation (it will work despite the errors) Refer to NVIDIA troubleshooting guide: https://documentation.ubuntu.com/server/how-to/graphics/install-nvidia-drivers/index.html sudo ubuntu-drivers autoinstall && sudo reboot might fix the issue. 3. Set up the FormAI-v2 dataset: This project builds on the FormAI-v2 dataset, which provides vulnerable C code samples generated by large language models. Make sure to place FormAI-v2.json in a directory named FormAI-v2/ in this project root. This file is required to sample real vulnerabilities and generate prompts. To get FormAI-v2.json: Navigate to the project root directory of the vulnerable-code-generation project first. Clone the repository: git clone https://github.com/FormAI-Dataset/FormAI-dataset/ Extract the classification files: 7z x FormAI-dataset/FormAI-v2-classification.7z.001 This command will automatically handle both parts (.7z.001 and .7z.002) and extract the FormAI-v2.json file. Move the generated FormAI-v2.json to FormAI-v2/ folder mkdir -p FormAI-v2 mv FormAI-v2.json FormAI-v2/ FormAI-v2.json contains metadata and vulnerability annotations for each file in the dataset. Each entry in the FormAI-v2.json file includes: filename: name of the C source file (also the model name in the file name). vuln_line and vuln_column: the location of the vulnerability in the code. violation: the violated ESBMC property (i.e., the vulnerability type). snippet: a short code excerpt showing the vulnerable region. cyclomatic_complexity: a complexity score used for bucketed analysis. 4. Install ESBMC: Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo https://github.com/esbmc/esbmc and add the binary to your environment's path: # Download the latest esbmc-linux.zip from ESBMC GitHub Releases. Get the latest `esbmc-linux.zip` from the ESBMC GitHub Releases. unzip esbmc-linux.zip -d esbmc # Add ESBMC to your PATH # Replace "path/to" with the actual location where you unzipped ESBMC echo 'export PATH=path/to/esbmc/linux-release/bin:$PATH' >> ~/.bashrc # or echo 'export PATH=path/to/esbmc/bin:$PATH' >> ~/.bashrc # You might need to set permissions chmod +x path/to/esbmc/linux-release/bin/esbmc # or chmod +x path/to/esbmc/bin/esbmc # Reload your shell configuration source ~/.bashrc Verify ESBMC installation: esbmc --version 5. Installation Verification Tests: Generate dynamic prompts: python prompt-generation/dynamic/dynamic_prompt_generation.py Expected output: Saved 2040 prompts to prompt-generation/dynamic/dynamic_prompts.json Integrate ESBMC integration by extracting sample code: python analysis/esbmc/extract_code.py The script will prompt for the model result file path. Enter the model output file path relative to the model-output directory (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json). This will create compilable C files in the c-code directory, confirming that the code extraction and compilation validation pipeline is working correctly. Installation tested on: Ubuntu 24.04 TLS with Python 3.x, using Kaggle GPU P100 accelerator for model inference. Running the Experiments and Generating Results Dynamic Prompting (Zero-shot) - prompt-generation/dynamic Generate the Prompts: To generate the dynamic prompts, run the following script: python prompt-generation/dynamic/dynamic_prompt_generation.py Expected stdout: Saved 2040 prompts to prompt-generation/dynamic/dynamic_prompts.json Run the Models on Prompts The models are run using Jupyter notebooks for Qwen2, Mistral, and Gemma, located in the kaggle-notebooks directory. These notebooks are executed in Kaggle environment. Steps to run the notebooks: Sign in to Kaggle Import the desired notebook (Qwen2, Mistral, or Gemma) from the original project kaggle-notebooks directory Click "Edit" to enter edit mode Go to Settings and select P100 accelerator Add your Hugging Face token: Create token at (https://huggingface.co/settings/tokens)[https://huggingface.co/settings/tokens] Set permission: "Read access to contents of all public gated repos you can access" Set variable hf_token in the Kaggle notebook Upload prompts from the right sidebar: Upload prompt-generation/dynamic/dynamic_prompts.json or prompt-generation/reverse/reverse_prompts.json Name the dataset "new-prompts" Or modify the prompt_path variable in the notebook if using a different dataset name The outputs will be saved to results/<model_name>_results.json. The model_outputs directory contains these JSON files. Note: CUDA/cuDNN warning messages during execution are harmless and can be ignored. Sample 100 Prompts (For Temperature Experiment) If you wish to run the temperature experiment with a sample of prompts, run: python prompt-generation/dynamic/sample_prompts.py Expected stdout: Saved 100 prompts to .../prompt-generation/dynamic/dynamic_prompts_temp_sample.json Reverse Prompting (One-Shot) - sampled_code Generation - prompt-generation/reverse Extract Vulnerable Code from FormAI-v2: python prompt-generation/reverse/extract_vulnerable_code.py Expected stdout: [done] Extracted 742252 vulnerable files into .../prompt-generation/reverse/formai_all_vulnerable_code The script takes input from ./FormAI-v2/FormAI-v2.json. It filters valid, verified vulnerable code samples, categorizes them by vulnerability type and cyclomatic complexity, and saves the source code as .c files in ./prompt-generation/reverse/formai_all_vulnerable_code/<vuln_type>/<cc_bucket>/. Get a Sample of Vulnerable Code: python prompt-generation/reverse/get_sample_code.py The script takes input from ./prompt-generation/reverse/formai_all_vulnerable_code/, randomly samples up to 5 .c files per subtype and cyclomatic complexity bucket, and saves them to ./prompt-generation/reverse/sampled_code/<vuln_type>/<subtype>/<cc_bucket>/. It organizes samples by vulnerability type, subtype, and CC bucket, resolving subtype-to-folder mappings where needed. Generate Reverse Prompts Using Gemini API (or any other LLM): Please create a Gemini API key at (https://aistudio.google.com/app/apikey)[https://aistudio.google.com/app/apikey] and add it to the generate_reverse_prompts.py file. Then run the following: cd prompt-generation/reverse python generate_reverse_prompts.py cd ../.. You can also change the Gemini setup and call_llm(prompt) function with a commercial LLm of your choice. However, this could impact the final results. Run ESBMC Analysis Run ESBMC Analysis After collecting model outputs, extract C code with: python analysis/esbmc/extract_code.py Enter the model result filename (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json): Before running the script, ensure that the model outputs are placed in the following path format: ./model-outputs/<dynamic|reverse>/<Model>/<JSON File> (e.g., ./model-outputs/reverse/qwen2/qwen2_reverse_prompting_full.json). Then, provide the relative path from ./model-outputs as the input to process the file (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json). The output will be stored in the ./c-code directory. The script saves the extracted .c source files and temporary .out binaries generated during compilation. Files are saved under the following structure: ../../c-code/<mode>/<model>/ Example (based on user input reverse/qwen2/qwen2_reverse_prompting_full.json): ../../c-code/reverse/qwen2/ After attempting to compile each .c file, the script removes: Any .c files that fail to compile All temporary .out binary files generated during compilation Only compilable .c files are retained in the output directory. Run ESBMC on Extracted Code python analysis/esbmc/esbmc_analysis.py Before running this script, ensure that you have already generated the extracted .c files by completing the previous step. This script analyzes the extracted C files using ESBMC. It expects the same relative model result path provided in the previous step (e.g., reverse/qwen2/qwen2_reverse_prompting_full.json) as input. It performs the following actions: Locates the corresponding .c files under ./c-code/<dynamic|reverse>/<Model>/, based on the input path. Runs ESBMC on each .c file with a 15-second timeout, using verification flags such as --incremental-bmc and --overflow-check. Writes raw ESBMC output logs to ./analysis/esbmc-reports/<mode>/, one .txt file per C file. Generates two JSON summaries in ./analysis/vuln-analysis-summary/<mode>/: A detailed summary with ESBMC results, matched metadata (e.g., vulnerability type, prompt), and violated properties if any. An executive summary with counts of each result type and, if applicable, a breakdown by temperature. Timeouts, parsing errors, and verification failures are identified and reported. Only .c files in the extracted code directory are processed. Full Analysis Once the C code is extracted and analyzed, run the following scripts for full analysis: cd analysis # For dynamic prompting: python full_analysis_dynamic_prompting.py # For reverse prompting: python full_analysis_reverse_prompting.py cd .. These will generate detailed analysis summaries, which are stored in the ./final-summary directory. Generating Reports and Visualizations full_generation_visual.py and heatmap.py use the data from ./final-summary and generate figures 5 and 6 in the paper respectively. The results in tables 1 to 4 are manually taken from ./final-summary. ./final-summary includes two files, ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt. Each file is organized by model and research question. The RQ1: Vulnerable‐code and correct‐vuln counts part in ./final-summary/dynamic_prompting_full_analysis.txt and ./final-summary/reverse_prompting_full_analysis.txt contains results for Table-1 The RQ2: By intent (absolute counts) parts contains results for Table-2. The RQ3: By form (absolute counts) parts contains results for Table-3. The RQ4: Incorrect Vulnerability Generation Patterns parts contains results for Table-4. Notes The "total_prompts": 2040, "code_generated": 2034, "compilable_code": 1711 attributes in the executive summary need to be added manually by checking the results from ./analysis/esbmc/extract_code.py (this part was not automated). For reverse prompt generation, Gemini API key is required. You can use any other LLM API, but be aware that results may differ significantly due to different model architectures and training data. Outputs and Results All intermediate and final results—including model generations, C code files, ESBMC reports, and summaries—are included in the replication release v1.0. These outputs are required to reproduce the tables and findings in the paper and are not tracked in the main branch due to their size.