Replication package for "When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench"

Replication package for the manuscript "When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench" (Sungbin Im, aSSIST; submitted to Empirical Software Engineering, November 2026). Contents: the gate-injection harness and the four evaluator implementations (code/); all 2,160 schema-validated policy_event_trace records from tau2-bench airline and retail episodes with collision-free public identifiers (data/traces/); all evaluator outputs (native scoring, LLM judge, policy-aware judge, boundary evaluator), judge and gate prompts with model version identifiers, cost and judge ledgers, and derived analysis tables (data/analysis_out/); the annotation kit (rubric v1.3, blinded episode packets, both annotators' original label files under neutral names, recalibration records, adjudication worksheet with written justifications, final reference-standard labels) (annotation/); supplementary material S1–S4 and the claims table verified against the manuscript (docs/); and a manifest with SHA-256 checksums for every file. A top-level README describes the layout and the license (code MIT; data and documents CC BY 4.0). Benchmark: sierra-research/tau2-bench at commit 363133a, used unmodified. Full simulation transcripts and the private packet-to-episode mapping are not included.