Replication package for "On Coding Agent Issue Localization Accuracy An Exploratory Study"

Replication Package

On the Accuracy of Issue Localization by Coding Agents

1. Overview

We empirically evaluate how accurately three open-weightLLM-based coding agents identify the software entities thathuman developers modify to resolve real-world issues. Thestudy spans 10 Apache Java projects and 2,441 issues, withmetrics computed at three granularities: package, class, andmethod.

The pipeline reconstructs each issue's historical pre-fixstate, invokes each agent on the issue description, andcompares the agent-modified entities against the human groundtruth using Precision, Recall, F1, Accuracy, and MCC.

2. Repository Structure

.
├── README.md # this file├── CHANGELOG.md                                           # All notable changes are documented in this file
├── requirements.txt # Python dependencies
├── data/
│ ├── raw/ # SQuaD dataset (input CSV files)
│ ├── interim/ # filtered issues per project (parquet)
│ │ ├── study_projects.csv # List of the 10 selected projects
│ │ ├── pilot_issues__<project>.parquet # Initial pool of issues per project (before size filtering)
│ │ └── pilot_final__<project>.parquet # Sampled issues per project (after size filtering)
│ ├── processed/
│ │ ├── ground_truth/ # human-fix entities per issue
│ │ │ └── <project>/
│ │ │ │ └── <issue>.json # Package/class/method entities modified by the human in the HIFC
│ │ ├── agent_outputs/ # raw patch.diff + metadata per (model, issue)
│ │ │ └── <model>/
│ │ │ └── <project>__<issue>/
│ │ │ ├── patch.diff # Generated patch
│ │ │ ├── run_metadata.json # Timing, exit code, timeout flag
│ │ │ ├── stdout.log # OpenCode standard output
│ │ │ └── stderr.log # OpenCode standard error
│ │ ├── agent_entities/ # Parsed entities from each agent patch (output of stage 07)
│ │ │ └── <model>/
│ │ │ └── <project>/
│ │ │ └── <issue>.json # Package/class/method entities modified by the agent
│ │ └── metrics/ # P/R/F1 per (model, project, granularity)
│ │ ├── _common/ # Issue keys completed by all three models for a given project (output of stage 09)
│ │ │ ├── common_issues__<project>.csv
│ │ │ └── all_metrics.csv # Union of all per-issue metrics across every model, project, and granularity
│ │ ├── _global/ # Final aggregated results (output of stage 10)
│ │ │ ├── per_model.csv # Macro-averaged P/R/F1/Acc per (model, granularity)
│ │ │ ├── per_project.csv # Per-project breakdown
│ │ │ └── cross.csv # Cross-model averages
│ │ └── <model>/ # Per-model metrics (one folder per LLM)
│ │ ├── per_issue__<project>.csv # Per-issue metrics computed by stage 08 (all valid runs)
│ │ └── per_issue_filtered__<project>.csv # Subset restricted to issues completed by all three models
│ ├── repos/ # cloned Apache repositories
│ │ └── apache#<project>/
│ └── workspaces/ # transient git worktrees (created at runtime)
├── scripts/
│ ├── 00_download_csv_SQuaD.sh # Download SQuaD source files
│ ├── 01_filter_and_link.py # Build the DuckDB database from SQuaD files
│ ├── 02_clone_repos.py # Clone Apache repositories listed in the file data/interim/study_projects.csv
│ ├── 03_select_pilot_subset.py # Select the issues for each project and create the pilot_issues_<project>.parquet file
│ ├── 04_extract_ground_truth.py # For each issue in pilot_issues_<project>.parquet, compute the human ground truth
│ ├── 04b_filter_pilot_for_size.py # [Not used] Filters selected issues and creates the `pilot_final_<project>.parquet` file
│ ├── 05_run_one_issue.py # Manual single-shot agent run for one (project, issue) pair
│ ├── 06_run_pilot.py # Orchestrate the agent runs over all issues in pilot_final__<project>.parquet
│ ├── 07_extract_aifc_entities.py # Parse agent patch with tree-sitter
│ ├── 08_compute_metrics.py # Compute TP/FP/FN/TN, P/R/F1, Accuracy, MCC per issue
│ ├── 09_filter_common_issues.py # Identify the intersection of valid issues across all 3 models
│ ├── 10_aggregate_metrics.py # Aggregate metrics across projects and models
│ ├── run_multi_project.py # orchestrator
│ └── count_divergent_cases.py # Count structurally divergent cases (potentially functionally equivalent)
├── llm_selection/
│ ├── artificialanalysis.csv # LLM ranking from artificialanalysis.ai
│ ├── llmstats.csv # LLM ranking from llmstats.com
│ ├── opencompass.csv # LLM ranking from OpenCompass leaderboard
│ ├── llm_selection.py # Aggregates the three rankings and produces the final ranked list
│ └── final_ranking.csv # Final ranked LLMs; top 3 open-weight models are used in the study
└── project_selection/
├── all_java_projects.csv # Candidate pool of Java projects from SQuaD
├── list_projects.py # Selects Java projects from SQuaD; writes all_java_projects.csv
├── projects_selection.py # Ranks candidates by 14 metrics; writes the two CSVs below and study_projects.csv
├── projects_at_least_9_median.csv # Projects above the median on ≥9 of 14 metrics
├── projects_at_least_9_q3.csv # Projects above the third quartile on ≥9 of 14 metrics (used for the study)
└── count_final_issues.py # Computes Cochran's sample size from the number of filtered issues

3. Requirements

System

OS: Linux (Ubuntu 22.04 or later recommended)

Python: 3.10 or later

Git: 2.43 or later

Python dependencies

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Agent harness

The Ollama Cloud API key must be configured in two places:

1. As an environment variable (used by the pipeline scripts):

export OLLAMA_API_KEY=""

2. In the OpenCode configuration file (used by the agent harness):

# Edit OpenCode's config file
mkdir -p ~/.config/opencode
cat > ~/.config/opencode/opencode.json << 'EOF'
{
"provider": {
"ollama-cloud": {
"api_key": "<your_key>"
}
}
}
EOF

Replace <your_key> with your Ollama Cloud API key in bothlocations.

The three open-weight LLMs evaluated in the paper are pulled byOllama at runtime:

ollama-cloud/glm-5:cloud

ollama-cloud/kimi-k2.5

ollama-cloud/qwen3.5:397b

4. LLM Selection

The three LLMs evaluated in the study (GLM-5, Kimi-K2.5,Qwen3.5:397B) were selected through a rank-aggregationprocedure implemented in llm_selection/:

Inputs. Three public leaderboards ranking open-weightLLMs on coding-related tasks, collected on March 2026.Each leaderboard provides an ordered list of models frombest to worst:

artificialanalysis.csv (artificialanalysis.ai)

llmstats.csv (llmstats.com)

opencompass.csv (OpenCompass leaderboard)

Aggregation. llm_selection.py combines the threerankings and produces final_ranking.csv.

The top-3 open-weight LLMs from final_ranking.csv areused to instantiate the three agents evaluated in the study.

To reproduce the selection:

python llm_selection/llm_selection.py

5. Project Selection

The 10 Apache Java projects evaluated in the study wereselected through a two-step procedure implemented inproject_selection/:

Step 1 — Candidate pool. list_projects.py queries theSQuaD database and retains every Java project on Apache JIRAwith a valid commit/issue linkage and a non-trivial number ofclosed bug-fix commits. The output is all_java_projects.csv.

Step 2 — Ranking. projects_selection.py evaluates eachcandidate against 14 size- and maturity-related metrics drawnfrom SQuaD. Two ranked subsets are produced:

projects_at_least_9_median.csv: projects above the medianon at least 9 of 14 metrics (lenient ranking)

projects_at_least_9_q3.csv: projects above the thirdquartile on at least 9 of 14 metrics (strict ranking)

The strict ranking is used to select the 10 top-rankedprojects analyzed in the study, written todata/interim/study_projects.csv.

Step 3 — Sample size. count_final_issues.py reads thefiltered issue counts per selected project and computes thetarget sample size using Cochran's formula (95% confidence,5% margin of error). The output guides the sampling stage ofthe main pipeline (03_select_pilot_subset.py).

To reproduce the selection:

python project_selection/list_projects.py --language Java --output all_java_projects.csv
python project_selection/projects_selection.py
python project_selection/count_final_issues.py --input data/interim/study_projects.csv

6. Execution Order

The pipeline is organized as a sequence of numbered stages.The early stages (00–04) prepare the data, the centralstage (run_multi_project.py) executes the agents on eachproject, and the final stages (09–10) aggregate theresults.

6.1 Full pipeline (recommended)

Step 1 — Data preparation

bash scripts/00_download_csv_SQuaD.sh

python scripts/01_filter_and_link.py

python scripts/02_clone_repos.py

python scripts/run_multi_project.py \
--projects-file data/interim/study_projects.csv \
--skip-04b \
--only-stages "03_select_pilot" "04_ground_truth" \
--continue-on-error

Step 2 — Agent runs

The orchestrator run_multi_project.py invokes one LLM at atime over all selected projects. Run it once per model:

for model in "ollama-cloud/glm-5:cloud" \
"ollama-cloud/kimi-k2.5" \
"ollama-cloud/qwen3.5:397b"; do
python scripts/run_multi_project.py \
--projects-file data/interim/study_projects.csv \
--model "$model" \
--skip-prep \
--skip-04b \
--timeout 600 \
--continue-on-error
done

For each (model, project) pair, the orchestrator runs in order:

06_run_pilot.py: invokes the agent on every issue inpilot_final__<project>.parquet and savespatch.diff + run_metadata.json intodata/processed/agent_outputs/<model>/<project>__<issue>/

07_extract_aifc_entities.py: parses each agent patch andextracts the modified entities at package, class, andmethod granularity

08_compute_metrics.py: computes TP/FP/FN/TN, Precision,Recall, F1, Accuracy, and MCC per (issue, granularity),and writesdata/processed/metrics/<model>/per_issue__<project>.csv

Each agent run is capped at a 600-second timeout. If a run exceeds the timeout,the same script must be restarted with the timeout parameter increased to 1200 seconds.Validity filtering (exit code 0 and not timed out) is applied at metric-computation time.

Step 3 — Cross-model aggregation

After all three models have completed, restrict the analysisto the intersection of valid issues, then aggregate:

python scripts/09_filter_common_issues.py \
--models ollama-cloud/glm-5:cloud \
ollama-cloud/kimi-k2.5 \
ollama-cloud/qwen3.5:397b

python scripts/10_aggregate_metrics.py \
--models ollama-cloud/glm-5:cloud \
ollama-cloud/kimi-k2.5 \
ollama-cloud/qwen3.5:397b

The aggregated CSVs in _global/ correspond to the numbersreported in Table 1 of the paper.

6.3 Running a single issue (debugging)

To experiment with a single (project, issue, model)combination — useful for testing prompts or harness behavior:

python scripts/prova_05_run_one_issue.py \
--project "apache#hudi" \
--issue HUDI-990 \
--model ollama-cloud/kimi-k2.5

This script does not write to the final metrics; its outputlives in data/processed/agent_outputs/ and can be inspectedmanually.

6.4 Running a single project

To run a single project across all stages (without iteratingover all 10), pass --projects instead of --projects-file:

python scripts/run_multi_project.py \
--projects "apache#hbase" \
--model ollama-cloud/glm-5:cloud
--skip-prep \
--skip-04b \
--timeout 600 \
--continue-on-error

7 Divergent Cases Analysis

The script count_divergent_cases.py quantifies the number of structurally divergentcases in the localization results, defined as issues for which the agent simultaneouslyintroduced modifications outside the human ground truth (fp > 0) and missed entitiesthat the developer actually touched (fn > 0). Structural divergence is identifieddeterministically from the confusion-matrix counts and does not require manual inspection.

These cases are relevant for the construct-validity discussion of the study: a subsetof them may correspond to functionally equivalent implementations that overlap-basedmetrics (Accuracy, Precision, Recall, F1, MCC) cannot capture, since the agent may resolvethe issue through differently named or structurally distinct entities. Counting themtherefore provides a conservative upper bound on how many evaluations could potentiallybe re-classified once functional equivalence is assessed through manual inspection orsemantic analysis — a step left to future work.

Input. The script expects a single CSV file with one row per (model, project, issue,granularity) combination, containing the columns: project, model, issue_key, granularity, n_human, n_agent, tp, fp, fn, tn, accuracy, precision, recall, f1, mcc.

In our pipeline, per-issue metrics are produced by 09_filter_common_issues.py and stored underdata/processed/metrics/<model>/per_issue_filtered__apache__<project>.csv, i.e. one fileper (model, project) pair. Before running the divergent-cases analysis, these files mustbe concatenated into a single consolidated CSV. This can be done with a one-liner:

awk 'FNR==1 && NR!=1 { next } { print }' \
data/processed/metrics/ollama-cloud__*/per_issue_filtered__apache__*.csv \
> data/processed/metrics/_common/all_metrics.csv

The resulting all_metrics.csv is the input to count_divergent_cases.py

Output. A summary printed to standard output reporting (i) the total number andpercentage of structurally divergent rows, and breakdowns by (ii) model,(iii) granularity, (iv) project, and (v) the cross-tabulation model × granularity.Optionally, the script exports the subset of divergent rows to a separate CSV forfurther manual inspection.

awk 'FNR==1 && NR!=1 { next } { print }' \
data/processed/metrics/ollama-cloud__*/per_issue_filtered__apache__*.csv \
> data/processed/metrics/_common/all_metrics.csv

python scripts/count_divergent_cases.py \
data/processed/metrics/_common/all_metrics.csv

python scripts/count_divergent_cases.py \
data/processed/metrics/_common/all_metrics.csv \
--output data/processed/metrics/_common/divergent_rows.csv

8 Changelog

All notable changes to this dataset are documented in this file.

[2.0.0] - 2026-06-11

Fixed

Replaced corrupted files in llm_selection/ folder with correct versions

[1.0.0] - 2026-05-29

Added

Initial dataset release

9 License

This replication package contains material distributed undermultiple licenses, depending on its origin:

Component
License
File

Code (Python scripts, shell scripts)
MIT License
SCRIPT_LICENSE

Derived data and aggregated results
CC-BY 4.0
DATA_LICENSE

Third-party material (Apache projects, SQuaD, OpenCode, LLMs)
Retained under original licenses
NOTICE.md

See NOTICE.md for full attribution and third-party licenseinformation.