Certified Splitting: the derivation and measurement archive

This is the evidence behind Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization. It is deposited on its own so that the archive has an identifier independent of the manuscript and of the library the manuscript measures. What changed in v2. This version accompanies the manuscript's second arXiv version. The programs that emit the frozen-transfer and campaign tables spell their headers "percent" and "block-comment" as the manuscript now does, the frozen inventory caches carry the keys those programs compute, and the archive adds the modules the gate now imports, among them those of the companion paper on finitely verifiable segmentation. The programs, their comments and their messages use American spelling throughout, which renames a few internal identifiers (among them the class Materializer and the functions _normalize and minimized_size) and changes no decision; the frozen study's two supply-share lines now end in "percent", and the three inventory caches were derived again from scratch after these edits changed the source digests in their keys, every decision coming out identical. Every figure the manuscript quotes is unchanged and still held by the gate. What is here. The archive keeps the layout of the tree it was cut from, so every relative path the programs use resolves as it does there. paper-drafts/certified-splitting/ holds data/, with every emission the manuscript quotes and the three committed inventory caches, and run-artifact.sh, the runner of the threaded scanner. explorations/certification/ holds the programs that derive the emissions and machine-check the theorems. The exploration directory includes the programs used by this paper and additional work for other papers. Only the modules exercised by the verification stages form this paper's executable checks. paper-drafts/certified-recovery/checker/ is the reference model of maximal-munch scanning the programs import, and paper-drafts/certified-recovery/data/r6/ holds the four campaign corpora the campaign instrument reads. scripts/check-paper4.sh is the gate the author runs in the repository, shipped for reference, and verify-all.sh at the root runs those of its stages that do not rebuild the manuscript. Verifying it. One ledger covers every file here except itself. To check the bytes: sha256sum -c manifest.txt To rerun the claims, sh verify-all.sh at the root. It needs python3, sha256sum and curl. It checks the ledger, fetches the one benchmark corpus the programs read, regenerates the emissions it can from the programs and holds each to its committed copy byte for byte, and checks the ledger again at the end, so a run that rewrote a cache would not report success. Whatever it cannot run it names with its reason. What is not here. The benchmark corpus is twitter.json, byte-identical to jsonexamples/twitter.json at simdjson's release v3.10.1. It is third-party content and is not redistributed; verify-all.sh fetches it from that pinned release and holds it to its SHA-256, 30721e49... The vocabulary measurements read the byte-level GPT-2 tokenizer's vocabulary and merge table, which are not redistributed either. Placed in a directory llm-vocabs beside the archive's root, under the names and digests verify-all.sh prints, they are held to those digests and the vocabulary emissions rerun; absent, those stages are named as not run. The threaded scanner is the munch library, release v2.1.0, archived at doi:10.5281/zenodo.23026012. run-artifact.sh names the commit and the digest of each probe source and refuses to run against anything else. Given a build of that library at the pinned commit, run-artifact.sh reruns it, and the author gate reruns the library's own recomputation of the emissions; the recorded result is data/artifact-stats.txt. Everything in this deposit is the author's own work and is released under CC BY 4.0. The one program of the exploration directory that adapts third-party material, splice_theorem.py, keeps it under the notices in explorations/certification/THIRD_PARTY_NOTICES.md; the program belongs to another paper, and this deposit carries neither it nor that file.