Replication package for evaluating local language models as triage judges for synthetic structured API policy evidence
收藏资源简介:
This version contains the corrected replication package for Evaluating Local Language Models as Triage Judges for Synthetic Structured API Policy Evidence. It preserves historical evidence separately and adds a complete new evaluation of thirteen local model checkpoints under an explicitly recorded synthetic policy contract. The corrected benchmark removes label-bearing case identifiers from every model prompt and repairs thirteen ineffective or overlapping interventions. The archive contains 4,880 newly scored model judgments: 3,120 main-comparison judgments, 480 additional prompt-ablation judgments, 720 second-generator-seed judgments, 180 controlled-challenge judgments, 120 repeat judgments, and 260 post hoc output-budget judgments. A separate 260-call timing experiment provides accepted timing measurements after classification-phase latencies were withdrawn for system sleep interruptions. Warm-up calls are excluded. Materials include synthetic evidence packets, exact prompts, complete model responses, checkpoint and runtime provenance, executable generator and predicate checks, strict output parsing, statistical analyses, empirical figure and table generation, pinned dependencies, reproduction instructions, and integrity hashes. Analyses include 5,000 dimension-stratified bootstrap resamples with 90% percentile intervals, paired comparisons with Holm correction, explicit confusion and abstention counts, risk-weight and prevalence sensitivity, and separately labeled post hoc output-format and token-budget diagnostics. Results concern a declared synthetic contract and do not establish production effectiveness or independently validated explanation quality. Author-generated code is MIT licensed; synthetic data and generated outputs are CC BY 4.0. Model weights are not redistributed and retain their respective licenses. Manuscripts, response letters, journal templates, and submission notes are excluded. The original version remains available as historical provenance; original and corrected observations must not be pooled.



