lemondecider-decision-index-results

# LemonDecider v3 — corrected recursive question-bisection evaluation **Public Decision Index 0.3: 60.03.** Complete: true. 140178 freshly inferred requests, 140148 ok and 30 unsupported. This supersedes the 57.33 no-split serving run for release use; it does not erase that historical measurement. Model: inclusionAI/LemonDecider (immutable v3 revision pending). Scores: runs/lemondecider-v3-chunked-public03/scores.json. Publication-named service: code/serve_312k.py (SHA-256 42239af50d3509570a53e20cf857a49a12ba11d2f4fb747950eae89b8f1a6e8f). Kit commit: 25c6ec7b6b5f44a216c63c3a700de3a9c6d6699d. Four independent PPU-ZW810E workers, BF16/SDPA, batch size one, native temperature 1.0026751894886363. ## Serving contract Text only. Maximum 8192 tokens per chunk; preserve full state and every option. Short requests remain joint; overlong question lists are recursively bisected. Oversized singletons reject the whole request before inference. Answers retain original IDs/order; token usage includes repeated state. No reread or truncation. 488 requests were split (779 additional forwards), recovering 488 previous refusals. See EVALUATION_CORRECTION.md for the defect and mandatory regression checks. This Dataset embeds the publication-named service as an audit/reproduction reference; no third repository is required. The HF model repository MUST use this matching service rather than the older no-split bridge before associating this score with a release. Model weights/config/tokenizer remain in the model repository, not this Dataset. ## Validation and limitations Scorer JSON files are byte-identical. Only response.model display labels in the published predictions were normalized; all other JSONL bytes are unchanged. Per-shard metadata and sanitized provenance are retained; top-level environment and status files are explicitly post-hoc summaries. No suite prompts/labels, weights, raw server logs or credentials are distributed. 442 excluded requests were not inferred. No placeholder results are added. Internal ledger registration remains blocked by its completed+excluded row-count assumption; this does not change the official scorer's complete=true result. This is a local public score, not an official private Full score or hardware gate. Original input-source terms still apply; no blanket dataset license is invented. Before upload fill in the immutable model revision and confirm access/rights. ```bash python verify_dataset.py python tests/test_chunking.py ``` The second command needs FastAPI/Uvicorn/Pillow import dependencies but no GPU. ## Publication naming normalization The original private evaluation archive is retained unchanged. In this public copy, 140148 response.model labels are normalized to `lemondecider-checkpoint`. This is a display identifier, not a new inference run or a new weight revision. Answers, probabilities, timings, IDs and statuses are unchanged. Before/after hashes and the transformation scope are in provenance/PUBLICATION-TRANSFORM.json. Code/tests now use LemonDecider names and the published model's environment-variable conventions. This code is not byte-identical to the historical service, but its plan/predict method ASTs are identical. Historical source hashes remain evidence, not assertions that these publication copies are byte-identical.