Reproducibility and Extended Evaluation of SOAPFL: A Standard OperAting Procedure for LLM-Based Method-Level Fault Localization

This artifact package provides the full reproducibility and extended evaluation materials for SOAPFL: (A Standard Operating Procedure for LLM-Based Method-Level Fault Localization). The goal of this package is to support transparent validation of our findings and enable future researchers to replicate, re-run, or extend our experiments under the same or updated conditions. The package contains: (1) the complete experimental results and logs obtained from our reproducibility rerun of SOAPFL on Defects4J v1.2.0 under the original configuration (GPT-3.5), (2) modified and extended scripts used to evaluate SOAPFL on Defects4J v1.4.0 with GPT-4o, and (3) additional scripts and outputs used for cross-language evaluation on the BugsInPy benchmark (Python). We also provide the prompt templates, configuration files, benchmark setup instructions, and automation scripts required to reproduce the execution pipeline. To improve usability and reproducibility, the artifact includes environment requirements, execution commands, intermediate outputs (e.g., extracted test failure logs and localization traces), and Top-k evaluation summaries. The package is organized to clearly separate the original reproduction setting from the extended evaluations, allowing researchers to independently reproduce each research question (RQ1 and RQ2). This artifact is intended to serve as a reusable evaluation baseline for future studies in LLM-based fault localization, benchmark extension, and cross-language debugging research.