Replication package: Measuring Specification Determinacy with Fenced, Specification-Only Reimplementation (RFC 8785, TRACE v0.2 and a Signed-Agreement Draft)
收藏资源简介:
Replication package for the manuscript "Measuring Specification Determinacy with Fenced, Specification-Only Reimplementation: RFC 8785, TRACE v0.2 and a Signed-Agreement Draft" (draft 5, 27 October 2026; the PDF is included). The study measures how much of a verifier's verdict is fixed by its specification text: fresh language-model agents (claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5; two runs each) implement each specification from its text alone under a mechanically audited fence, log every point the text did not decide, and are run with the reference implementation over corpora of real inputs and clause-targeted mutants (2,034, 2,816 and 9,960 inputs); every disagreement is attributed to a clause and classified by two language-model raters and, on a blind sample of 50 clusters, by a human third rater, Sankalp Gilda, whose answers are reported as received. Subjects: RFC 8785 (control), TRACE v0.2 (agentrust-io/trace-spec at commit 473045fd) and a2a-agreement v1.1 (the author's draft, commit 33d2df82), plus an exploratory amendment of the latter measured with six further implementations. Contents: PROTOCOL.md (design file with a correction block, chronology and incident log), fences/ (the exact specification files given to the implementers, with SHA-256 sums), runs/ (24 counted and 3 incident implementations with their DECISIONS.md and READ_LOG.md), corpus/, results/ (per-implementation verdicts, the two oracles for S2, analysis, clusters, rater packets, both rounds of model ratings, the third rater's packet, answers and cluster mapping with the three-rater agreement, fence audit, the two adversarial reviews, the moderation and odds notes), decisions/ (the agreement-statistics decision record, fixed and anchored to a public ledger before the third rater's answers were read), tools/ (every script that computes and cross-checks the figures in the paper), paper/ (LaTeX source of drafts 4 and 5). Version 5 adds the third rater (Section 3.5 of the manuscript); nothing reported in version 4 was changed. The TRACE findings were reported to the project as agentrust-io/trace-spec issue #591. Licence: Apache-2.0 for code and data; the specification copies keep their own licences (RFC 8785: IETF Trust; TRACE: Apache-2.0, agentrust-io; a2a-agreement: Apache-2.0). Not pre-registered; see PROTOCOL.md and Section 2.5 of the manuscript. Authorship note (31 October 2026): at his request, Sankalp Gilda is no longer listed as an author. He is acknowledged as the human third rater. The PDF and the LaTeX source in this version were made before that change and still name him as second author; the next version will correct them.



