Beyond Heuristics? Rethinking Targeted Unit Test Generation in the Era of LLM Agents

# Beyond Heuristics? Rethinking Targeted Unit Test Generation in the Era of LLM Agents This repository contains the replication package for the paper **"Beyond Heuristics? Rethinking Targeted Unit Test Generation in the Era of LLM Agents"**, submitted to **ESEM 2026** (under double-blind review). ## Overview We conduct a comprehensive evaluation of LLM agents for **targeted unit test generation** — the task of generating a JUnit test suite that reproduces a specific bug at a given location in the codebase. We evaluate **3 agent scaffolds** (Claude Code, OpenHands, OpenCode) × **3 LLMs** (Claude 4.5 Sonnet, GPT-5, DeepSeek-V3.2) on a benchmark of **210 real-world Java bugs** from **66 open-source projects**, and compare against state-of-the-art traditional tools (UnitCon, EvoSuite, Randoop). **Key findings:**- The best combination (Claude Code + GPT-5) achieves an **89.5% success rate**, surpassing UnitCon (49.6%) by 39.9 percentage points.- LLM agents demonstrate three core advantages: direct code comprehension, execution-guided iterative refinement, and flexible internal API access via reflection.- Even open-source models (DeepSeek-V3.2) outperform all traditional tools at ~$0.04 per instance.- The primary failure modes are **Insufficient Exploration** (41.2%) and **Inadequate Reasoning** (25.6%). ## Repository Structure ```├── run_evaluation.py # Main evaluation orchestration script├── build_image.py # Docker image builder for benchmark cases├── docker/ # Docker configurations│ ├── java_base/ # Base Java image (multi-JDK support)│ ├── jdk_17/ # JDK 17 image│ ├── jdk_21/ # JDK 21 image│ └── jdk_25/ # JDK 25 image├── benchmarks/ # 210 real-world bug instances│ ├── metadata.json # Case metadata (Java version, build commands)│ ├── <case_name>/ # Each bug instance│ │ ├── submit.py # Test submission script│ │ └── repo-src/ # Buggy source code│ └── ...└── lipics-v2021-sample-article.tex # Paper source (LaTeX)``` ## Requirements - Python 3.8+- Docker- One or more of the evaluated LLM agents: [OpenCode](https://opencode.ai), Claude Code, or OpenHands > **Note:** Before running the evaluation, you must configure the target LLM (e.g., Claude 4.5 Sonnet, GPT-5, or DeepSeek-V3.2) as the **default model** in your agent's configuration. Each agent scaffold (Claude Code, OpenHands, OpenCode) reads its model setting from its own configuration file or environment variable; refer to the respective agent's documentation for details. ## Usage ### Evaluate LLM agents on the benchmark ```bashpython run_evaluation.py benchmarks/ -f opencode -t 900 -r 5``` Arguments:- `benchmarks/` — Path to the benchmarks directory- `-f`, `--framework` — Agent scaffold: `opencode`, `claude`, or `openhands`- `-t`, `--time-budget` — Time budget in seconds per case (default: 900)- `-r`, `--repetitions` — Repetitions per case (default: 1)- `-n`, `--case-count` — Randomly select N cases (default: all) ### Build a specific benchmark Docker image ```bashpython build_image.py <case_name> -b benchmarks/``` ## Benchmark Our benchmark contains **210 real-world Java bugs** from **66 projects**, including Defects4J, BEARS, and recent bug datasets. Each instance includes: - The buggy source code (`repo-src/`)- Build configuration (defined in `metadata.json`)- A submission script (`submit.py`) for compiling and running generated tests in a Docker container The benchmark spans multiple exception types, with `NullPointerException` being the most prevalent (126 cases).