The Substitutability Illusion: Replication Package for a Model-Substitutability Audit of a Scaffolded LLM Pipeline

Replication package for an empirical audit of cross-model substitutability at the outputs of an independently developed, enterprise-style scaffolded LLM research prototype. The study evaluates 8 frozen synthetic engineer personas across 5 hosted language models from 4 providers, with 5 accepted repetitions per persona-model cell, yielding 200 accepted end-to-end runs. Each run executes a four-call scaffolded pipeline. Cross-model behavior is compared against each model's own run-to-run variability using preregistered agreement metrics and a primary ±0.10 equivalence margin. The package contains the frozen synthetic fixtures and hash locks; model/version mapping and decoding configuration; scaffold configuration fragments; analysis code for the M1–M6 metric suite; a 10,000-replicate run-level hierarchical bootstrap with seed 20260803; margin and leave-one-persona-out sensitivity analyses; parse-tolerant rating and opportunity-type sensitivity analyses; complete accounting of 280 schema-validation events; per-run and per-call provenance for all 200 accepted runs; and the derived tables and figures reported in the associated manuscript. The public core releases categorical structured outputs sufficient to independently recompute all six M1 metrics. Free-text model-generated extracts required to re-derive M2 and M3 are withheld pending a redistribution determination. Complete raw provider responses were not retained, and end-to-end pipeline re-execution is outside the artifact scope because the base synthesis system prompt was stored by reference rather than captured. The included self-verification script confirms the 8 × 5 × 5 study structure, frozen fixture hashes, M1 point estimates, the primary 5 rejected / 5 inconclusive / 1 demonstrated equivalence tally, parse-tolerant sensitivity results, and complete accounting of all 280 schema-validation events.