Improving Repair Agents with Cause-Chain Diagnoses

This record holds the complete evaluation results for the paper Improving Repair Agents with Cause-Chain Diagnoses. FORECAST is a debugging approach that gives repair agents a diagnosis of the failure before they start. From the executions of passing and failing tests, it infers a cause chain. This is a short sequence of execution features, such as conditions, data flow, values, and calls, that separate the failing from the passing runs. FORECAST appends this chain to the issue an agent receives, so it works with any agent without modifying it. The evaluation covers all 300 instances of SWE-bench Lite (test split). Five repair agents (mini-swe-agent, SWE-agent, AutoCodeRover, RepairAgent, and Agentless) ran with two models, Claude Haiku 4.5 (claude-haiku-4-5-20251001) and Qwen3.5 27B (qwen3.5:27b). Each agent ran in two modes. In the baseline mode it received the issue, and in the forecast mode it received the issue plus FORECAST's diagnosis. This gives 20 runs of 300 instances each. Over all agents, the share of resolved issues rises from 48.5% to 70.9% with Claude Haiku 4.5 and from 31.2% to 56.1% with Qwen3.5. Contents forecast-results.tar.zst (320 MB, about 54 GiB and 908,277 files extracted) contains: README.md, which documents the layout of the archive and how to reproduce the analysis results.json, the per-instance results that all numbers in the paper are computed from diagnoses/, FORECAST's diagnosis for each instance (cause chain with discriminability, the text appended to the issue, spectrum-based fault localization, and FORECAST's step report with times) <model>/<mode>/, every repair run: the submitted patch, the agent's log, every LLM call with messages, responses, tokens and cost, the upstream agent's own trajectory, and the SWE-bench harness verdict with per-test status The files are archived byte for byte as the evaluation wrote them. The archive contains no API keys and no timestamps from the runs. It leaves out the SWE-bench harness's test logs, FORECAST's execution traces and relevance trees, and the schedulers' logs. Usage sha256sum -c SHA256SUMS tar --zstd -xf forecast-results.tar.zst SHA256SUMS is in evaluation/results/ of the code repository. The research questions are answered from results.json by evaluation/scripts/analyze.py, and results.json can be rebuilt byte-identically from the extracted archive. Links Code: https://github.com/smythi93/forecast Package: https://pypi.org/project/forecast-repair/