The Methodeutic Harness on SWE-bench Pro: public-split run, receipts, and an oracle-access correction
收藏资源简介:
A solo, fully-receipted run of a methodeutic agent harness across all 728 eligible SWE-bench Pro public-split instances (694/728 = 95.3% under the official grader, every losing run committed and re-gradable from source-only diffs). Published with a correction in the open: on this public split the gate read the held-out FAIL_TO_PASS tests as its stopping signal, iterating the method against the verifier, the move SWE-bench's held-out protocol forbids. The 95.3% is therefore an oracle-availability ceiling (an implement-only loop floors near 50%; gate access to the visible tests raises it to about 96%), not a harness lift over bare models. The mistake is named as a Type III error (a precise answer to the wrong question) and it motivated the companion mechanism experiment that measures the harness where no visible oracle exists. Archived for the record, the nulls and the mistake included. Note: this repository is dual-licensed, code under AGPL-3.0 and content under CC BY-SA-NS (see LICENSE.md); pick the matching license at the Zenodo release step.



