ScholarHound analysis artifacts: frozen rules, model outputs, scoring records, and the complete benchmark-construction screening census (preprint v0.4)
收藏资源简介:
Complete analysis bundle supporting the preprint Accountability without an Oracle: ScholarHound for LLM-Produced Scientific Commitments, and Why Claim-Level Gold Is the Binding Constraint (v0.4). Contents: Frozen rules — SciFact quorum gate V2 with freeze timestamp 2026-07-02T08:09:36Z, and digest gate V1 as a separate frozen artifact. The freeze predates every held-out prediction. Sampling manifests — 60-item development set; 30-item holdout under seed 2026070201; balanced strict-16 selection manifest with public task hash 03697e003b502243d53f0179139314442822c7e5138e1a8eb963b6586f1aa0eb. Raw model outputs and parser status — all SciFact runs across five model families, and all eight provider-condition digest runs (128 rows), with request timestamps and execution-surface identifiers retained verbatim, including the model_id=qoderwork record treated as unverified in the paper. Scoring artifacts — per-item aggregate votes, validation-and-scores record, and final evaluation report, sufficient to reproduce every reported number including the execution-surface sensitivity analysis. Complete screening census — the record behind the paper's central measurement: roughly 27,000 screened records across four public sources yielding 16 executable items. Three discarded packet designs are included by intent, each with its stated rejection reason, because the census is only auditable if the rejected stages are inspectable. Two categories are deliberately excluded and documented in EXCLUSIONS.md: publisher-copyrighted full text (replaced by DOIs, locators, and the verbatim spans actually shown to the models), and Retraction Watch records (cited by identifier and checksum rather than redistributed, consistent with their licence). Integrity: every file is listed in SHA256SUMS. Verify with shasum -a 256 -c SHA256SUMS. Reuse constraints are stated in MANIFEST.md. In particular, the balanced strict-16 packet is a completed diagnostic set and must not be used to tune digest gate V1, and reviewer-derived scoring receipts must remain in their separate directory because flattening the layout destroys the blinding.



