遇见数据集

Multi-judge rescoring of small-LLM response aggregation: code and data

收藏
Zenodo2026-10-29 更新2026-11-05 收录
官方服务:

资源简介:

Corrected release. The Claude judge scores in v1.0.0 are invalid because the scoring wrapper replaced failed or unparseable judge calls with a score of 3 without recording which scores were substitutions. In this version every answer is rescored by seven judges from four vendor families (Claude Haiku 4.5, Sonnet 4.6, Opus 4.7; Gemma 4 26B; Mistral Small 3.2 24B; Qwen3.5 9B; Qwen2.5-VL 32B) with an identical 1 to 5 rubric, and failed judge calls are recorded as empty cells; no default value is substituted. The record contains 2,800 generated answers from seven response-aggregation configurations (B0 to B6) over a 400-query multilingual test set (Korean, English, German, Japanese), with latency and cost; 19,600 judge scores (three empty); and the scripts for rescoring, the generalizability study, rank-reversal counts, and inter-judge agreement. B0 and B6 derive from the authors' own earlier systems; B1 to B5 are approximate configurations of published methods within the authors' harness. Source question text is withheld; every item identifier is published.

提供机构:
Zenodo
创建时间:
2026-10-29
二维码
社区交流群
二维码
科研交流群
商业服务