Artifacts for Paper Submission

Artifacts for Paper Submission to the 34th USENIX Security Symposium

This repository contains the source code and dataset associated with our submission to the 34th USENIX Security Symposium. Below are detailed instructions for setting up the environment, installing dependencies, and reproducing the results presented in the paper.

Quick Start

1. Setup Environment

Create and activate a Conda environment with the required Python version:

conda create -n RACGSecurity python=3.10
conda activate RACGSecurity

Install the necessary Python packages:

pip install pytorch openai transformers crystalbleu ollama rank-bm25 sklearn numpy

2.Set API Key

To use LLMs, specify your API key in `src/configs.py`. Below is an example configuration:

"gpt-4o": {
"base_url": "https://api.openai.com/v1/",
"key": "Your API key here"
}

For local models such as Llama-3, CodeLlama, and DeepSeek-Coder-V2, they are implemented using the [ollama framework](https://ollama.com/). Refer to their respective official documentation for setup instructions.

Reproducing Results

1. Generate Embeddings for the Retriever

Run the following script to create embeddings for the retriever:

python ./src/gen_embedding.py

2. Scenario I

Use the following command to generate code with `gpt-4o` under a `one`-shot setting, using the `JINA` retriever and a poisoning number of `5`:

python src/Scenario_I.py --model gpt-4o --poisoned_num 5 --shots_num 1 --retriever JINA --dataset_path ./dataset/ReposVul.jsonl --embeddings_path ./embeddings

3. Scenario II

Similar to Scenario I, but this scenario uses `poisoned_proportion` to measure poisoning quantities:

python src/Scenario_II.py --model gpt-4o --poisoned_proportion 0.6 --shots_num 1 --retriever JINA --dataset_path ./dataset/ReposVul.jsonl --embeddings_path ./embeddings

4. Evaluation

Evaluate the security of the LLM-generated code by running:

python ./src/eval_security.py --generated_code_path "GENERATED_CODE_PATH"

Replace `"GENERATED_CODE_PATH"` with the actual path to the generated code.