LocalizeAgent Reproducibility Package

Replication Package: An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization

This replication package contains the necessary code and instructions to run the experiments described in the paper, "An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization".

Prerequisites

Python 3.12.2

Required Python packages:

pip install anthropic

pip install openai

pip install google-generativeai

pip install tqdm

pip install joblib

pip install javalang

pip install numpy

Installation

1. Clone the repository or extract the replication package.2. Install the required Python packages

Running the Experiments

Generating LLM Responses

To generate responses from language models (LLMs) like Claude, OpenAI, or Gemini, use the run.py script with the appropriate arguments.

 python3 run.py --MODEL_NAME --output ./openai/ --path PATH_TO_EVALUATION_DATASET

For example,

 python3 run.py --openai --output ./openai/move_m --path EvaluationDataset/move_m/single

--openai: Use the OpenAI API (alternatively, use --claude or --gemini for other LLMs).

--output: Specify the directory to save the LLM responses.

--path: Provide the path to the directory containing the Java files.

You can also use additional arguments like --limit to limit the number of files to process, --context to add context to the prompt, and --log_file to specify a log file path.

Evaluating LLM Responses

To evaluate the LLM responses by comparing function names extracted from file IDs and response texts, use the eval_llms.py script.

python3 run.py --eval --path /openai/move_m/llm_responses.json

--eval: Enable evaluation mode.

--path: Provide the path to the JSON file containing the LLM responses.

Running the Evaluation Script

The run_eval.sh script is provided to automate the evaluation process for multiple LLM responses. It loops through a predefined set of paths and executes the evaluation command for each path.

bash ./run_eval.sh

Customization

You can modify the source code files (eval_llms.py, llm_interfaces.py, prompt_generation.py, and utils.py) to customize the behavior or add new features as needed.

Additional Notes

The "EvaluationDataset" directory is already included in the replication package.

The output directory specified with the --output argument will be created if it doesn't already exist.