Artifacts for Paper Submission

This repository contains the source code and dataset associated with our submission to ACM CCS 2025. Below, you'll find instructions for installing dependencies, setting up the environment, and running the provided tools.

Installation and Requirements

To evaluate the effectiveness of CodeGuarder, please first install the benchmark CyberSecEval by following the instructions in the official CyberSecEval repository.

Next, install the Python dependencies required to run our tools:

pip install -r requirements.txt

Running CodeGuarder

To run CodeGuarder, ensure that the API key for your LLM is correctly set in the `config.py` file. For DeepSeek-Coder and CodeLlama, follow the instructions on the Ollama website to deploy the models locally.

To reproduce the results reported in the paper, use the following commands:

Generating Data for the Standard Scenario

#### Without CodeGuarder (Baseline)

python src/Baseline_Standard.py --output_path "./dataset/Standard.json"

#### With CodeGuarder (Defense)

python src/Defense_Standard.py --output_path "./dataset/Standard.json"

### Evaluating CodeGuarder Under the Standard Scenario

Make sure CyberSecEval has been cloned into the `CyberSecEval` directory.

cd CyberSecEval/CybersecurityBenchmarks

python -m CybersecurityBenchmarks.benchmark.run \

    --benchmark=instruct \

    --prompt-path="./datasets/instruct/Standard.json" \

    --llm-under-test="OPENAI::{MODEL_NAME}::{MODEL_KEY}::{BASE_URL}"

Replace `{MODEL_NAME}`, `{MODEL_KEY}`, and `{BASE_URL}` with the appropriate values for your LLM. For example, if using OpenAI's GPT-4, `BASE_URL` is the API endpoint. For **DeepSeek**, use `https://api.deepseek.com/v1`.

For other scenarios, modify the corresponding `Baseline_*.py` and `Defense_*.py` files. All generated data will be saved in the `dataset` directory.

Security Evaluation

Evaluation results are stored in the `result` directory. For instance, to evaluate the effectiveness of CodeGuarder under the Standard scenario with DeepSeek-Coder, run:

python src/sec_eval.py --result_path "./output/std_def_DS-Coder.json"