CertBasis-Bench: FAA Part 25 special-conditions benchmark (exploratory release)

A benchmark of FAA Part 25 certification-basis tasks built from published FAA special conditions (Federal Register, 1994-2026, via the SC-25 dataset) and eCFR Part 25: T1 trace a requirement to regulation, T2 the Part 25 sections a special condition addresses (first-of-kind vs re-issue strata; masked inputs), T5 amendment awareness. Exploratory release: dev and non-sealed test items with gold, per-run scores for five open-weight and five closed models plus a decision-protocol arm, precedent baselines, the scoring harness and the analysis code. The 141 sealed hold-out items are withheld (salted SHA-256 commitment; see the seal note). Inputs are US Government works in the public domain. Novelty labels are automatic and AI-made: AI-only, no human verification. Repository: https://github.com/ai-lab-industrial/certbasis-bench. Paper: https://doi.org/10.5281/zenodo.23911638. Source dataset: SC-25 0.1.0 (https://doi.org/10.5281/zenodo.23911695). The 141 sealed hold-out items are withheld; a salted SHA-256 commitment is published. The seal protects only against exposure through our own evaluation pipeline: no sealed item was sent in any evaluation run. One logged exception: during the pilot and rating stages on 2026-11-02, the masked inputs, publication dates and unlabelled candidate section lists of 8 sealed T2 items (no answer key) reached GPT-6.1 Sol (OpenAI) and Claude Opus 5.5 (Anthropic); for 4 of them the novel-feature description and public Federal Register link also reached Claude Opus 5.5. These 8 items are flagged as provider-exposed (exposure log events E1-E3; the public log gives the count) and are excluded from, or reported separately in, any future confirmatory evaluation of OpenAI or Anthropic models. The seal does not protect against reconstruction from public sources: the Federal Register text is public, and SC-25 makes the sealed items' answers derivable. A future confirmatory test must therefore use a prospective hold-out (special conditions issued after this release and after each model's training cutoff) or a fresh seal. The commitment still proves which items were sealed, and when. Funding: This work received no specific grant from any funding agency. Compute (cloud model API credits) was provided in kind by Industrial AI Lab. Competing interests: G. Gonçalves is Vice President of Industrial AI Lab (non-profit) and owns SmartHow, a company that provides AI solutions for industry. SmartHow could in future build or sell AI tools for certification or compliance work of the kind this benchmark evaluates. No SmartHow product is evaluated. AI use: Claude Code (Anthropic; Claude Opus 5.5) wrote the pipeline, scoring code and figures and drafted the synthetic T1 items; the SC-25 families and labels were produced by an AI agent. AI-only; no human verification. AI is not an author. Data: CC0-1.0 (LICENSE-DATA). Code: Apache-2.0 (LICENSE).