Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

Replication Package for: Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks This repository contains the scripts and setup to evaluate the generation of vulnerable code by open-weight Large Language Models (LLMs) in response to various prompts. We use dynamic prompting (zero-shot) and reverse prompting (one-shot) techniques to guide the models (Qwen2, Mistral, and Gemma) to generate C code that contains specific vulnerabilities. Experiment Setup OS and Environment The experiments were run on Kaggle Notebooks using default settings. The analysis scripts were run on Ubuntu 24.04 TLS Python 3.x environment with the following dependencies: transformers accelerate einops bitsandbytes These dependencies are listed in the requirements.txt file. Models Qwen2-7B-Instruct Mistral-7B-Instruct Gemma-7B-Instruct These models were loaded using the Hugging Face transformers library with quantization options for efficient inference. GPU/Hardware The experiment was conducted on Kaggle using their GPU P100 accelerator. Dataset: FormAI-v2 This project builds on the FormAI-v2 dataset, which provides vulnerable C code samples generated by large language models. To use the dataset in reverse prompting, download the accompanying metadata file: FormAI-v2.json: This JSON file contains metadata and vulnerability annotations for each file in the dataset. Each entry in the FormAI-v2.json file includes: filename: name of the C source file (also the model name in the file name). vuln_line and vuln_column: the location of the vulnerability in the code. violation: the violated ESBMC property (i.e., the vulnerability type). snippet: a short code excerpt showing the vulnerable region. cyclomatic_complexity: a complexity score used for bucketed analysis. You can obtain this file and the full dataset of .c files by cloning or downloading the repository: git clone https://github.com/FormAI-Dataset/FormAI-dataset Make sure to place FormAI-v2.json in a directory named FormAI-v2/ in the project root for the reverse prompting scripts to work. This file is required to sample real vulnerabilities and generate reverse prompts conditioned on actual code characteristics. Experiment Commands Dynamic Prompting (Zero-shot) Generate the Prompts: To generate the dynamic prompts, run the following script: python prompt-generation/dynamic/dynamic_prompt_generation.py Run the Models on Prompts: The models are run using the respective Jupyter notebooks for Qwen2, Mistral, and Gemma. The generated prompts are provided as input to these notebooks. Sample 100 Prompts (For Temperature Experiment) If you wish to run the temperature experiment with a sample of prompts, run: python prompt-generation/dynamic/sample_prompts.py Reverse Prompting (One-Shot) Extract Vulnerable Code from FormAI-v2: python prompt-generation/reverse/extract_vulnerable_code.py Get a Sample of Vulnerable Code: python prompt-generation/reverse/get_sample_code.py Generate Reverse Prompts Using Gemini API (or any other LLM): Add your Gemini API key to the generate_reverse_prompts.py file and run: python prompt-generation/reverse/generate_reverse_prompts.py You can also change the Gemini setup and call_llm(prompt) function with a commercial LLm of your choice. However, this could impact the final results. ESBMC Setup Install ESBMC: Download and extract the latest esbmc-linux.zip from the ESBMC GitHub repo (github.com/esbmc/esbmc) and add the binary to your environment's path. Run ESBMC Analysis After collecting model outputs, extract C code with: python analysis/esbmc/extract_code.py --input path_to_model_output_file Run ESBMC on Extracted Code python analysis/esbmc/esbmc_analysis.py --input path_to_extracted_code Full Analysis Once the C code is extracted and analyzed, run the following scripts for full analysis: python full_analysis_dynamic_prompting.py python full_analysis_reverse_prompting.py These will generate detailed analysis summaries, which are stored in the final-summary/ directory. Notes The "total_prompts": 2040, "code_generated": 2034, "compilable_code": 1711 attributes in the executive summary need to be added manually by checking the results from extract_code.py (this part was not automated). For reverse prompt generation, Gemini API key is required. You can use any other LLM API, but be aware that results may differ significantly due to different model architectures and training data. Dependencies To set up the environment, simply install the necessary dependencies by running: pip install -r requirements.txt Outputs and Results All intermediate and final results—including model generations, C code files, ESBMC reports, and summaries—are included in the replication release v1.0. These outputs are required to reproduce the tables and findings in the paper and are not tracked in the main branch due to their size. Final Remarks This repository provides a comprehensive framework for evaluating the security risks associated with the use of LLMs in generating code. The techniques can be adapted to other tasks involving code generation, vulnerability analysis, and prompt engineering for LLMs.