Language Advantage Is a Function of the Executor: criteria self-audit and undecidability audit in cross-language code generation by LLM coders (timetable-ab)
收藏资源简介:
v1.6.3 (2026-11-01, documentation only). Every confidence interval in this release now carries its resampling provenance (bootstrap, 20000 iterations, seed 20261101). Point estimates do not move with the seed; interval endpoints move in the third decimal (for example the lower bound of the inline language ratio spans 0.825–0.829 across seeds). Tables state the seed, so a reader reproducing a number knows which draw it came from. v1.6.2 (2026-11-01, documentation only). The headline figures in the public-facing pages were replaced with current values. index.html, one-pager.html and knock.md no longer lead with 1.93x; the first figure a reader sees is now 1.008x (inline) with its interval [0.825, 1.116] crossing 1.0, and the abstraction effect is stated as 1.200x (a 17% saving, measured independently in both languages). The withdrawn figure is still named explicitly as withdrawn — nothing was deleted. No measurement, criterion or dataset changed in this version. v1.6.1 (2026-11-01, same day as v1.6.0). A spec gap was closed and the package was audited. Spec gap closed, not excluded. Invariants #31 (unknown verb) and #32 (empty command string) had always required err BADARG, but the specification never described those two inputs, so the same artefacts could be scored two ways (2/12 with all 36 invariants, 5/12 with the two excluded). Instead of excluding the invariants, the requirement is now written into tasks/v3.spec.md section 4.1, the --exclude-out-of-spec switch is deleted, and a new freeze record (FREEZE-v1.6.1-amendment.txt) carries the post-amendment hashes. Consequence, recorded rather than hidden: Common Lisp first-pass returns to 2/12 and the 5/12 figure reported in v1.6.0 is withdrawn. Package audit. Six scans over 918 files: no credentials of any kind; thirteen files redacted to remove absolute paths containing a user name; one stray compiled artefact deleted; no file above 1 MB; seven genuine freeze-hash mismatches documented (three historical FREEZE files describe the pre-amendment state and keep their original hashes, annotated with a pointer); and seven public documents that still presented withdrawn figures as results (1.93x and the "28-40% saving") now carry correction banners stating the current values. v1.6.0 (2026-11-01). Three changes, each of them narrowing what this dataset claims. (a) Criteria closure. The mutation score was finally reproduced (Common Lisp 83.3%, JavaScript 70.6%). The previously recorded 0% for Common Lisp was an artefact: check-scale.cjs lacked the WSL dispatch, so all 36 invariants crashed and the baseline was red. Eight surviving mutants — five per arm plus two found outside the sample — were reduced to seven behaviours, now pinned by new invariants K1-K7 in check-closure.cjs; each carries its spec line number and each is green on both baselines and on a legitimate alternative implementation. (b) Gate hardening. The regression gate used to run only check-scale.cjs, so deleting the new criteria left it green. It now checks configuration self-consistency, a criteria-set fingerprint, missing implementations and evidence-table provenance, and its behaviour is validated by eight tamper tests (A passes, B-G each fail on the intended reason, H restored passes). (c) Contract boundary. check-v3.cjs had been mixing two contracts in one file — invariants 1-36 belong to the 15-op spec, 37-56 to a 30-op variant — so every 15-op implementation scored 36/56 regardless of quality. The checker now refuses to run without an explicit denominator. Two invariants with no basis in the spec (#31 unknown verb, #32 empty command) are excluded under a new flag; excluding them moves Common Lisp first-pass from 2/12 to 5/12, which is reported as a second denominator rather than substituted silently. Add-n experiment. With n raised from 3 to 6 per arm under two token budgets, the language claim becomes an interval: inline JavaScript/Common Lisp = 1.008x [0.825, 1.116]; abstraction = 1.007x [0.879, 1.095]; both crossing 1.0. The abstraction effect is 1.200x [1.077, 1.442] in Common Lisp and 1.200x [1.072, 1.362] in JavaScript — two independent paths yielding the same number. The earlier "28-40% saving" is retracted; the measured value is 17%. A further observation: 6 of 12 Common Lisp cells fail to even load (unbalanced parentheses, reader errors, a variable called as a function) against 12/12 loading for JavaScript. Original proposition. When the executor is an AI large language model, does Lisp have a measurable language advantage? Eleven candidate levers were pre-registered and tested with 7 languages, 81 runs and 36 invariants, each invariant first proven able to go red before it was allowed to say "pass". This is a restructured version of v1.0. Conclusions are now reported with a three-way verdict — established / refuted / undecidable — and two files were added: CONDITIONS.md (the conditions required to decide the question, and which of them are missing) and experiments/timetable-ab/results.tsv (one machine-readable row per run; columns that need human judgement are left empty rather than guessed). Three-way result. Established (3): expansion-time rejection (19/19 invariants green with validation off vs. load-time failure with it on); zero ceremony cost (all three arms, 0/0 manual edits); live-image REPL saves characters (9028 → 5957). Refuted (3): S-expressions are not the most compact (CL 7506 vs JS 5933 characters at 5 ops); skill-pack intervention showed no main effect (0/6); at 15 ops the Common Lisp implementation is 1.93× the JavaScript one. Undecidable (6): every remaining lever, for the reasons below. Missing conditions, stated plainly. C1 no weak-executor channel — only a strong model was available; 1.5B–7B tiers do not run on the author's machine and no external quota was available, so the entire P>0 half of the question was never observed. C2 no exclusive machine or clock alignment — all timing data is void and the crossover point N* is undecidable. C3 a single model family — language effect cannot be separated from training-distribution effect. Two further items are incomplete rather than missing: C4 nine surviving mutants (coverage of silent-failure detection unknown) and C5 scale (≤30 ops, ≤1k lines; no extrapolation to larger projects). Three things this dataset does not do. It does not read "not detected" as "not present"; it does not extrapolate beyond the measured scale; it does not delete data — the timing columns, the retired line-count metric and all nineteen recorded failures of the measurement apparatus remain in the chronicle, annotated with their scope. Included: 81 run artefacts, 14 frozen specifications, the 36 invariants with their go-red proofs, mutation scores with sample-size bounds and the twelve mutation floors, CONDITIONS.md, results.tsv with RESULTS-SCHEMA.md, a 2318-line append-only chronicle, and CLAIMS.md — one row per claim, with the command that checks it, the file it writes and the boundary it holds under. Excluded: compiled binaries, build caches, machine-local temporary files, personal data. Intended use: replication of the measurement protocol; reuse of the mutation-testing gate as a way to audit one's own criteria; and an empirical basis for benchmark-validity work in LLM code generation.



