The pair test on HelpSteer2: an AI judge against a human rubric

Pre-registrations, raw judge outputs, analysis code and the exploratory record for §7 of the manuscript "Exhibiting a missing axis: a pair test for framework adequacy, with the cost signal as an open proposal" (Pavel Gurov). The pair test asks whether a framework assigns two cases to one class although they differ in what a reader can do with them. Here the framework is a helpfulness score on the HelpSteer2 dataset, and the pair is a factually wrong and a correct answer to the same prompt. Result: on 150 fresh contested prompts, an AI judge (Claude Sonnet 5.5) gave the wrong and the correct answer the same helpfulness score on every run in 29 cases (19.3%, 95% CI 13.3–26.0%); NVIDIA's human ratings under the same definition did so in none. A second judge (Claude Haiku 5.5) on the same prompts did so in 20 (13.3%, 95% CI 8.0–18.7%); the difference between the two judges is not distinguishable from zero. Both tests were pre-registered; two earlier exploratory stages are included and labelled as such. Contents:- phase1/: pair test with the human helpfulness rating as the framework, all 10,681 prompts (exploratory)- phase2/: pair test with the AI judge, 12 responses per request (exploratory)- confirmatory/: pre-registered test on 150 fresh prompts, one response per request- second_judge/: the same pre-registered test with Claude Haiku 5.5 on the same prompts (added in version 1.1)- runner/: the three pages that ran the judges, and a script that rebuilds them byte for byte- README.md, SHA256SUMS Each stage holds its plan (written before the data it concerns were analysed, with every later change logged), its report, its code and its outputs. All analyses reproduce the stored results byte for byte. HelpSteer2 is not redistributed; README.md gives download sources and SHA-256 hashes. Version 1.1 adds a pre-registered second judge (Claude Haiku 5.5) on the same prompts.