dpa-ase2026

# Distributional Program Analysis — Artifact Reproduction package for **"Distributional Program Analysis: Treating Large LanguageModels as Program Samplers"** (ASE '26, NIER Track). DPA treats an LLM as a sampler over programs for one specification, runs a propertyextractor on each sample, and aggregates the results into a per-invariant **survivalrate** — the fraction of samples in which the extractor infers that invariant. This artifact reproduces every number in the paper's *Artifact evidence* subsection.It does **not** contain the HumanEval+ pilot (164 tasks, *r* = 0.78, pass@164.6 / 71.3 / 75.0); that data will be released with the full study. ## Requirements Python 3.10+. No third-party dependencies. ## Reproducing the paper ```shmake # runs both targets belowmake smoketest # Table 1 and the filtering resultmake ablations # threshold sweep, random-filter baseline, analyzer cost``` Both targets are deterministic — the random baseline is seeded — so repeated runsproduce byte-identical output. | Paper claim | Command | Output || ----------------------------------------------------------------- | ------------------ | --------------------------------- || Table 1: eight facts with survival 20/20 … 17/20 | `make smoketest` | `runs/smoketest/posterior.json` || 7.7 invariants per sample; 17 of 20 kept; purity 100% vs. raw 85% | `make smoketest` | `runs/smoketest/result.json` || Threshold sweep: 100% for τ ≤ 0.85, then 94.4 / 89.5 / 85.0% | `make ablations` | `runs/ablations/result.json` || Random filter: fully correct pool in 10 of 10,000 draws (0.1%) | `make ablations` | `runs/ablations/result.json` || Analyzer cost 0.18 ms per sample | `make ablations` | `runs/ablations/result.json` || 17 distinct sources among 20 samples | `make ablations` | `runs/ablations/result.json` | Per-sample programs and their inferred invariants are written to`runs/smoketest/samples/` and `runs/smoketest/invariants/`. ## What the analyzer is `dpa/analyzer/predicate.py` is a **deterministic checker over a closed predicatevocabulary**, in the likely-invariant tradition of Daikon — *not* a static abstractinterpreter. Each candidate invariant is a `⟨kind, body⟩` pair with `kind` drawn from`type`, `length_relation`, `boundary`, `elementwise`, `postcondition`, and an optionalguard (`… when k > 0`). A fact is reported for a sample only if it holds on **every**probe input without raising, so a single exception suppresses it. Because the vocabulary is closed, canonicalization (`Invariant.canonical`) is exactnormalization rather than heuristic matching. The cost is expressiveness: anything thevocabulary does not contain is invisible to survival by construction. ## Layout ```dpa/analyzer/ property extractor over the closed predicate vocabularydpa/invariants/ invariant representation and canonical formdpa/posterior/ survival-rate aggregationdpa/apps/ DPA-filterdpa/eval/ smoke test, ablations, pass@1 / majority-vote metricsdpa/task/ the fixed 20-sample rolling_max suite and its oracleruns/ recorded output of both targets``` ## Known characteristics of the sample suite The suite is a **fixed fixture**, not a live LLM sampling run (`model_id: fixed-smoke-samples`). It holds **17 distinct sources across its 20 entries** —`s03_too_short_single`, `s04_missing_guard`, and `s09_pairs_only` are byte-identical to`s00_slice`. The paper reports this figure as a mild instance of the sample collapsethat assumption (A1) excludes. Changing the fixture will change the reported numbers;re-run `make` to regenerate them. Results are from one task. They show the mechanism works and is reproducible, not thatit generalizes.