When Verification Rejects Correct Answers: Reproducibility Data and Code

Overview This repository provides the datasets, model outputs, verification algorithms, reference annotations, and reproducibility scripts associated with the study entitled "When Verification Rejects Correct Answers: A Diagnostic Study of Japanese LLM Decision Pipelines." The study investigates the reliability of large language model (LLM) decision-making pipelines, with particular attention to the distinction between factual information extraction, deterministic decision-rule application, and evidence-based output verification. The primary objective is to examine how verification mechanisms may reject correct model outputs when their underlying document parsers cannot adequately handle unfamiliar linguistic expressions or document structures. Study Design The experiments used synthetic Japanese-language operational decision tasks involving four information elements: Communication availability Round-trip response time Decision deadline Permission to defer a decision Two locally deployed language models, Qwen 2.5 7B Instruct and Gemma 3 4B, were evaluated using direct decision generation and a pipeline combining LLM-based fact extraction with deterministic rule execution. Three verification approaches were investigated: V1: Formal, quotation, and internal-consistency verification V2: Semantic verification based on isolated evidence quotations V3: Contextual verification using source-document location and surrounding textual context The evaluation included internally constructed synthetic documents and a separate set of 24 constrained synthetic documents written by two external contributors. The external documents were independently annotated by two readers, and their reference annotations were finalized before model evaluation. Key Findings In the internal evaluation, V3 accepted 77 of 192 extraction outputs, compared with 54 accepted by V2, without accepting an incorrect fact-and-action combination in that dataset. However, when applied to externally written documents, both V2 and V3 rejected all 96 extraction outputs, including outputs whose extracted facts and decisions matched the reference annotations. Further analysis identified limitations in the verifier's document parser, particularly its dependence on predefined vocabulary and line structure and its failure to distinguish unrecognized information from genuinely missing information. These findings demonstrate that internal improvements in verification coverage do not necessarily generalize to unfamiliar document formulations. Repository Contents The reproducibility archive contains: Synthetic Japanese-language evaluation texts Anonymized externally written evaluation documents Reference fact annotations and deterministic decision rules Recorded outputs from the evaluated local LLMs Verification and scoring implementations Reproducibility scripts and audit materials Documentation of experimental conditions and evaluation procedures The included scripts support offline reproduction of the archived analyses without requiring new LLM inference. Scope and Limitations This repository supports a diagnostic investigation of LLM-based decision verification under controlled synthetic information constraints. The experiments do not establish clinical safety, medical decision-making effectiveness, compliance with NASA operational requirements, or reliable performance in real-world space missions. The externally written documents were created under predefined semantic constraints and should not be interpreted as a representative sample of naturally occurring Japanese documents. The reported findings concern the evaluated models, verification implementations, datasets, and experimental conditions. In particular, zero incorrect outputs accepted by a verifier that rejects every output must not be interpreted as evidence of successful safety assurance. Reproducibility The archived inputs, reference annotations, recorded model outputs, and verification scripts enable independent inspection and offline reproduction of the reported analyses. The repository is intended to support transparent evaluation, methodological scrutiny, and further research into evidence-grounded LLM decision verification, selective acceptance, and the generalization limits of rule-based checking systems.