遇见数据集

Language Advantage Is a Function of the Executor: criteria self-audit and undecidability audit in a seven-language, 81-run study with LLM coders (timetable-ab)

收藏
Zenodo2026-10-29 更新2026-11-05 收录
官方服务:

资源简介:

Original proposition. When the executor is an AI large language model, does Lisp have a measurable language advantage? Eleven candidate levers were pre-registered and tested with 7 languages, 81 runs and 36 invariants, each invariant first proven able to go red before it was allowed to say "pass". This is a restructured version of v1.0. Conclusions are now reported with a three-way verdict — established / refuted / undecidable — and two files were added: CONDITIONS.md (the conditions required to decide the question, and which of them are missing) and experiments/timetable-ab/results.tsv (one machine-readable row per run; columns that need human judgement are left empty rather than guessed). Three-way result. Established (3): expansion-time rejection (19/19 invariants green with validation off vs. load-time failure with it on); zero ceremony cost (all three arms, 0/0 manual edits); live-image REPL saves characters (9028 → 5957). Refuted (3): S-expressions are not the most compact (CL 7506 vs JS 5933 characters at 5 ops); skill-pack intervention showed no main effect (0/6); at 15 ops the Common Lisp implementation is 1.93× the JavaScript one. Undecidable (6): every remaining lever, for the reasons below. Missing conditions, stated plainly. C1 no weak-executor channel — only a strong model was available; 1.5B–7B tiers do not run on the author's machine and no external quota was available, so the entire P>0 half of the question was never observed. C2 no exclusive machine or clock alignment — all timing data is void and the crossover point N* is undecidable. C3 a single model family — language effect cannot be separated from training-distribution effect. Two further items are incomplete rather than missing: C4 nine surviving mutants (coverage of silent-failure detection unknown) and C5 scale (≤30 ops, ≤1k lines; no extrapolation to larger projects). Three things this dataset does not do. It does not read "not detected" as "not present"; it does not extrapolate beyond the measured scale; it does not delete data — the timing columns, the retired line-count metric and all nineteen recorded failures of the measurement apparatus remain in the chronicle, annotated with their scope. Included: 81 run artefacts, 14 frozen specifications, the 36 invariants with their go-red proofs, mutation scores with sample-size bounds and the twelve mutation floors, CONDITIONS.md, results.tsv with RESULTS-SCHEMA.md, and a 2318-line append-only chronicle. Excluded: compiled binaries, build caches, machine-local temporary files, personal data. Intended use: replication of the measurement protocol; reuse of the mutation-testing gate as a way to audit one's own criteria; and an empirical basis for benchmark-validity work in LLM code generation.

提供机构:
Zenodo
创建时间:
2026-10-29
二维码
社区交流群
二维码
科研交流群
商业服务