Empirical Evaluation of SOAPFL: An LLM-Based Framework for Method-Level Fault Localization

1. Overview This repository contains the experimental artifacts for the paper "Empirical Evaluation of SOAPFL: An LLM-Based Framework for Method-Level Fault Localization." The primary objective of this artifact package is to support the reproducibility, validation, and extension of the experimental findings presented in the paper. It provides the scripts, configurations, benchmark setup instructions, prompts, execution logs, and evaluation results required to reproduce the experiments and investigate the performance of SOAPFL across different LLM backends and programming languages. The experimental evaluation covers three settings: Reproducibility evaluation on Defects4J v1.2.0: Re-execution of the original SOAPFL configuration using GPT-3.5-turbo-16k. Extended evaluation on Defects4J v1.2.0: Evaluation of the modified SOAPFL configuration using GPT-4o. Cross-language evaluation on BugsInPy: Extension of SOAPFL to Python projects to investigate its applicability beyond Java. The package is designed to enable researchers to reproduce the reported results, inspect intermediate execution outputs, and extend the evaluation to other LLM backends or benchmark configurations. 2. Experimental Benchmarks The experiments use the following fault-localization benchmarks. 2.1 Defects4J v1.2.0 Defects4J is a benchmark of real-world Java software bugs commonly used in software testing and fault-localization research. The evaluation uses Defects4J v1.2.0, covering 395 bugs across six Java projects: Chart Closure Lang Math Mockito Time All Java benchmark experiments reported in this artifact use Defects4J v1.2.0. No other Defects4J version is required or used. 2.2 BugsInPy BugsInPy is a benchmark containing real-world bugs from Python projects. The cross-language evaluation uses 402 bugs from 11 Python projects. The extended SOAPFL implementation includes the necessary adaptations for Python project checkout, test execution, coverage collection, and source-code analysis. 3. Environment Requirements The following software and tools are required to execute the experiments. Component Requirement Operating system Linux (Ubuntu recommended) Python Python 3.12 or later Java OpenJDK 8 or a compatible version required by the selected Defects4J projects Defects4J v1.2.0 BugsInPy Required for Python benchmark experiments OpenAI API Valid API key Git Required for repository and benchmark management Note: Although the artifact supports a modern Python environment, individual benchmark dependencies may impose additional version constraints. Researchers should follow the dependency requirements of the corresponding benchmark and experiment configuration. 3.1 Defects4J Configuration Before running SOAPFL on Defects4J, install and configure Defects4J v1.2.0 according to its official installation instructions. The repository also contains a directory named Defects4J_mod, which provides modifications required by the experimental implementation. Apply the relevant files from Defects4J_mod to the Defects4J v1.2.0 installation before executing the Java benchmark experiments. Important: The Java experiments in this artifact are configured exclusively for Defects4J v1.2.0. 4. Repository Organization The repository contains the implementation, configuration files, benchmark adaptations, and experimental outputs required to reproduce the evaluation. The main components are: Directory/File Description run.py Main entry point for executing SOAPFL experiments. Config/ Configuration files specifying the experimental settings. camel/ LLM backend and related implementation components. Defects4J_mod/ Modifications required for the Defects4J experimental setup. BugsInPy/ Python benchmark integration and cross-language evaluation components, where applicable. results/ Experimental outputs and localization results, where included. The exact contents may vary depending on the released artifact version. The execution scripts and configuration files should be consulted for the complete set of available options. 5. API Configuration SOAPFL uses an OpenAI API backend to interact with the selected language model. Before executing an experiment, configure your OpenAI API key in: camel/model_backend.py Set the API key using the configuration mechanism expected by the implementation. Security recommendation: Do not commit API keys or other credentials to the repository. Use an environment variable or a local configuration file excluded from version control whenever the implementation supports it. Ensure that the selected model is available to your OpenAI API account before starting an experiment. 6. Running SOAPFL The main execution entry point is run.py. The general command format is: python3 run.py \ --config <CONFIG_DIR> \ --version <D4J_VERSION> \ --project <PROJECT> \ --bugID <BUG_ID> \ --model <GPT_MODEL_NAME> 6.1 Reproducing the GPT-3.5 Evaluation To reproduce the original SOAPFL configuration on Defects4J v1.2.0 using GPT-3.5-turbo-16k, execute: python3 run.py \ --config Default \ --version 1.2.0 \ --project Closure \ --bugID 26 \ --model GPT_3_5_TURBO This command illustrates the execution of SOAPFL on Closure-26. Replace the project name and bug ID to execute other benchmark instances. 6.2 Running the GPT-4o Evaluation The modified SOAPFL configuration supports evaluation using GPT-4o. Use the corresponding configuration directory and model identifier provided in the repository. The general command structure remains: python3 run.py \ --config <GPT4O_CONFIG> \ --version 1.2.0 \ --project <PROJECT> \ --bugID <BUG_ID> \ --model <GPT4O_MODEL_IDENTIFIER> The exact configuration name and model identifier should match those defined in the released implementation. 6.3 Configuration Files Additional configurations are available in the Config/ directory. These files specify the experimental settings used by SOAPFL. Researchers should select the configuration corresponding to the experiment they intend to reproduce. 7. BugsInPy Cross-Language Evaluation The artifact also includes the extended implementation and experimental materials for evaluating SOAPFL on the BugsInPy benchmark. Unlike the Defects4J experiments, this evaluation targets Python projects and requires the corresponding BugsInPy environment and dependencies. The Python implementation incorporates the necessary adaptations for: Checking out benchmark projects and bug versions. Executing the corresponding test suites. Collecting test coverage information. Analyzing Python source code and method-level changes. Generating the information required by the SOAPFL localization pipeline. Recording localization results for Top-k evaluation. Researchers intending to reproduce the Python experiments should first install and configure BugsInPy and then use the scripts and instructions provided in the corresponding artifact directory. 8. Experimental Results and Logs The artifact package provides the experimental results and execution materials used to support the findings reported in the paper. The released materials cover: Defects4J v1.2.0 with GPT-3.5-turbo-16k: Results from the reproducibility evaluation of the original SOAPFL configuration. Defects4J v1.2.0 with GPT-4o: Results from the extended evaluation using the modified SOAPFL configuration. BugsInPy: Results from the cross-language evaluation of the extended SOAPFL implementation. Where available, the execution records include: LLM prompts and generated responses. Intermediate outputs produced during localization. Test execution and failure information. Coverage and source-code analysis outputs. Method-level localization results. Top-1, Top-3, and Top-5 evaluation summaries. Runtime and API usage information. These materials allow researchers to inspect the execution process and independently examine the reported localization outcomes. 9. Evaluation Metrics The paper evaluates fault-localization performance using Top-k localization metrics. Top-1: The faulty method is ranked first. Top-3: The faulty method appears among the three highest-ranked methods. Top-5: The faulty method appears among the five highest-ranked methods. The reported results summarize the number of benchmark bugs for which the faulty method is successfully identified within each ranking threshold. The runtime and API-cost analyses additionally examine the execution overhead associated with different LLM backends. 10. Reproducibility and Extension This artifact is intended to support independent reproduction and further investigation of LLM-based method-level fault localization. Researchers can use the provided materials to: Reproduce the original SOAPFL evaluation on Defects4J v1.2.0. Investigate the effect of changing the LLM backend from GPT-3.5-turbo-16k to GPT-4o. Reproduce the cross-language evaluation on BugsInPy. Inspect prompts, LLM responses, and intermediate localization outputs. Compare SOAPFL with the baseline techniques reported in the paper. Extend the implementation to additional LLM backends, benchmarks, or programming languages. For reproducibility, researchers should retain the benchmark versions, configuration files, model identifiers, and execution settings used in each experiment. LLM-generated outputs may vary across independent executions, even when the same configuration is used. 11. Artifact Availability The experimental results and supplementary materials are available through Zenodo: SOAPFL Experimental Artifacts The Zenodo package provides the supplementary experimental data and replication materials associated with the paper. 12. Citation If you use this artifact in your research, please cite the associated paper: Empirical Evaluation of SOAPFL: An LLM-Based Framework for Method-Level Fault Localization. Please refer to the published paper or its official bibliographic record for the complete citation information. 13. Acknowledgements We acknowledge the developers and maintainers of Defects4J and BugsInPy for providing publicly available benchmark datasets that support empirical research in software testing and fault localization. We also acknowledge the authors of the baseline approaches used in the evaluation.