Multi-judge rescoring of small-LLM response aggregation: code and data
收藏资源简介:
Corrected release; supersedes v2.0.0 and v1.0.0. In v2.0.0 the Claude rescoring was run from the authors' project directory, where the command-line client also loaded a short operator note (about 300 tokens, no scoring instruction, naming the research topic) into the judge context. In v2.0.1 all Claude scores come from a repeated rescoring run from an empty directory, and a reply counts as a score only if its first line is a single digit from 1 to 5 (two Mistral Small replies that are not scores, previously read as 2 and 5, are now empty cells). The Claude judge scores in v1.0.0 are invalid because the scoring wrapper replaced failed or unparseable judge calls with a score of 3 without recording which scores were substitutions. The record contains 2,800 generated answers from seven response-aggregation configurations (B0 to B6) over a 400-query multilingual test set (Korean, English, German, Japanese), with latency and cost; 19,600 judge scores from seven judges of four vendor families (Claude Haiku 4.5, Sonnet 4.6, Opus 4.7; Gemma 4 26B; Mistral Small 3.2 24B; Qwen3.5 9B; Qwen2.5-VL 32B), five of them empty, so that the fully crossed analysed design is 397 queries and 19,453 scores; and the scripts for rescoring, the generalizability study, rank-reversal counts, and inter-judge agreement. B0 and B6 derive from the authors' own earlier systems; B1 to B5 are approximate configurations of published methods within the authors' harness. Source question text is withheld; every item identifier is published.



