Counterfactual Triage and Evidence-Gated Autonomy for LLM Testing Agents: Benchmark, Code and Data (anonymized for review)

Replication package for the paper "Confirming Before Acting: Counterfactual Triage and Gated Autonomy for LLM Agents in Broadband Gateway Testing" (under double-blind review). This version is anonymized for review. Contents benchmark_code_and_results.zip: open testbed of emulated OpenWrt gateways (QEMU and network namespaces), fault definitions, counterfactual triage engine, evidence-gated policy and healing actions, risk-bounded evidence gate, LLM baselines with all prompts, analysis scripts, aggregated results used in the paper, test suites (including an independent OpenWrt community suite), LLM annotations of 120 OpenWrt bug reports with the code-history check, and documentation (docs/): fault catalog; example LLM-written defect report; selection of the systematic regressions, with every excluded commit and its reason; harness notes. benchmark_runs_runs2.zip … runs8.zip: per-trial evidence, labels, intervention logs, LLM responses and policy decisions: runs2: main campaign; runs3: systematic regressions; runs4: replication; runs5: live agents; runs6: final runs; runs7: live risk-gate validation; runs8: community-suite detection. benchmark_runs_runs6_spare9_nowan.zip: quarantined trials affected by a spare-unit network fault, kept for transparency and not used in the analysis. RequirementsFirmware images are not included; they are the unmodified OpenWrt x86-64 images 24.10.8 and 25.12.5 from downloads.openwrt.org. Re-running the LLM conditions requires an Anthropic API key (environment variable ANTHROPIC_API_KEY). All recorded results can be inspected and re-analyzed without an API key. Start with README.md in benchmark_code_and_results.zip. The OpenWrt community test suite in testbed/suite51c/tests/community/ is third-party code distributed under its original LGPL-2.1 license (see LICENSE and UPSTREAM in that folder). Bug-report data are released by issue number only, without contributor names.