SpecDiff Artifact

SpecDiff — Artifact Specification-traceable differential testing of PDF engines. This artifact contains the ambiguity-mining pipeline, the template/generation + differential-testing code, and the 228 validated cross-engine divergences (each with the variant PDF, per-engine page renders, and per-engine extracted text). What is here artifact/├── README.md ← this file├── ENGINES.md ← the 9 PDF engines and exact versions│├── phase1_ambiguity_analysis/ ← Phase 1: mine ambiguity points from the spec model│ ├── spec_analysis.py ← mining + evidence-validation script│ ├── arlington_model/ ← Arlington PDF model (36 TSVs) — the input spec model│ └── results/│ ├── candidate.csv ← the 5,466 executable groups│ └── out_gray/ ← intermediate pipeline stages│ ├── 00_parsed/ ← raw Arlington rows parsed from TSVs│ ├── 01_syntax/ ← deterministic syntactic pattern matches│ ├── 02_llm/ ← LLM candidate proposals (per object)│ ├── 03_merged/ ← merged syntax + LLM candidates│ ├── 04_graph/ ← cross-object dependency graph│ ├── 05_stats/ ← mining counts and per-stage statistics│ └── 06_llm/ ← evidence-validated final candidates│├── phase2_generation/ ← Phase 2/3: fill template holes, render, diff│ ├── README.md ← templates, ambiguity kinds, and how holes are filled│ └── code/│ ├── gen_samples_v3.py ← template library + per-kind variant generators│ ├── build_ambiguity_templates.py ← compile per-ambiguity templates with seed pinning│ ├── run_diff_v3_round2.py ← run each variant across the 9-engine panel│ ├── cross_engine_report_v2.py ← group engines per variant; record render/text divergences│ └── mapping.py ← object → template-family mapping│├── engine_wrappers/ ← the 9-engine differential panel that run_diff_v3_round2.py drives│ ├── README.md ← layout, quick start, how the rest of the artifact uses this│ ├── WRAPPER_API.md ← the unified ParserWrapper interface, in full│ ├── test_wrappers.py ← self-contained smoke test (builds its own PDF, runs all 9)│ ├── parsers/ ← the 9 engine wrappers behind one interface│ └── src/ ← comparison metrics (SSIM render diff, text diff) the runners import│└── divergences_228/ ← the 228 validated divergences ├── INDEX.csv ← 228 rows: category, set, object, key, kind, div_type, severity, path ├── README.md ← layout of each set and on-disk conventions ├── direct_output_194/ ← 194 cases; objects that directly reach rendering / text extraction └── non_direct_34/ ← objects outside the direct-output families ├── data/ ← 34 cases: Signatures / Forms / Annotations / OptionalContent └── kept_manifest.csv The 228 divergences category count scope evidence per case direct_output 194 Color / Fonts / Images / Pages / Rendering / XObjects, and their sub-objects variant .pdf + 7 render .png + 9 extractor .txt (layout varies by case) non_direct 34 Signatures / Forms / Annotations / OptionalContent renders/ + samples/ + txt/ (per case/variant) total 228 divergences_228/INDEX.csv lists all 228 with category (direct_output / non_direct), object / key / kind, divergence modality (R = render-only, T = text-only, RT = both), severity, and on-disk path. Modality summary category R T RT total direct_output_194 82 94 18 194 non_direct_34 0 26 8 34 total 82 120 26 228 Engines Nine widely used open-source engines: seven render pages and extract text (Ghostscript, MuPDF, PDFBox, PDFium, PDF.js, Poppler, Xpdf) and two text-only (pdfminer.six, pypdf). Exact versions are in ENGINES.md. Render oracle: max SSIM drop > 0.10 introduced by a field edit, relative to the baseline. Text oracle: edge variant changes the cross-engine output grouping (whitespace-normalized). engine_wrappers/ is the code that actually drives the nine engines and compares their outputs; phase2_generation/code/run_diff_v3_round2.py imports it directly. Put engine_wrappers/ on PYTHONPATH to re-run the differential test and reproduce the per-engine renders/text shipped under divergences_228/: pip install pikepdf pypdfium2 pdfminer.six pypdf Pillow scikit-image PyMuPDF JPype1cd engine_wrappers/parsers && npm install && cd ../.. # PDF.js enginePYTHONPATH=engine_wrappers python phase2_generation/code/run_diff_v3_round2.py <variant.pdf ...> See engine_wrappers/README.md for the wrapper layout and engine_wrappers/WRAPPER_API.md for the full ParserWrapper interface. The three engines that shell out to external binaries (xpdf, pdfjs, pdfbox) need their paths passed explicitly; exact versions used in the paper are in ENGINES.md.