Multi-judge rescoring of small-LLM response aggregation: code and data
收藏资源简介:
Corrected release. The Claude judge scores in v1.0.0 are invalid because the scoring wrapper replaced failed or unparseable judge calls with a score of 3 without recording which scores were substitutions. In this version every answer is rescored by seven judges from four vendor families (Claude Haiku 4.5, Sonnet 4.6, Opus 4.7; Gemma 4 26B; Mistral Small 3.2 24B; Qwen3.5 9B; Qwen2.5-VL 32B) with an identical 1 to 5 rubric, and failed judge calls are recorded as empty cells; no default value is substituted. The record contains 2,800 generated answers from seven response-aggregation configurations (B0 to B6) over a 400-query multilingual test set (Korean, English, German, Japanese), with latency and cost; 19,600 judge scores (three empty); and the scripts for rescoring, the generalizability study, rank-reversal counts, and inter-judge agreement. B0 and B6 derive from the authors' own earlier systems; B1 to B5 are approximate configurations of published methods within the authors' harness. Source question text is withheld; every item identifier is published.



