said-rag-eval-2026/said-rag-eval-benchmark

SAID RAG评估基准（v1.1）是一个包含75个单元的基准数据集，专注于RAG（检索增强生成）管道评估输出，使用10个LLM（大语言模型）评判指标，旨在研究LLM评判的RAG评估中的无监督指标可靠性过滤。该数据集源自NeurIPS 2026提交的论文，用于分析LLM评判中的系统性偏见。数据集包含5个来源数据集（HotpotQA、MS MARCO、WikiQA、PubMedQA、FinQA），每个数据集采样100个问题，共涉及5个生成器、3个前沿评判模型和32个管道配置，总计约240,000个答案记录和240万个指标值。数据文件包括压缩的指标分数文件和原始答案文件（但未包含原始问题、真实答案或上下文文本，需用户自行从源数据集获取并基于sample_id进行连接）。数据集主要用于研究LLM评判偏见、基准测试无监督指标聚合过滤器，以及压力测试新的RAG评估方法。

A 75-cell benchmark of RAG pipeline evaluation outputs with 10 LLM-judged metrics, designed to study unsupervised metric reliability filtering for LLM-judged RAG evaluation. This dataset is an artifact accompanying a NeurIPS 2026 submission, focusing on analyzing systematic biases in LLM-as-a-judge evaluation. It comprises 5 source datasets (HotpotQA, MS MARCO, WikiQA, PubMedQA, FinQA) with 100 questions sampled per dataset, involving 5 generators, 3 frontier judges, and 32 pipeline configurations, totaling approximately 240,000 answer records and 2.4 million metric values. The data files include compact metric scores and raw answer files (excluding original questions, ground truth answers, or context texts, which users must obtain from source datasets and join using sample_id). It is intended for studying LLM-judge biases, benchmarking unsupervised metric-aggregation filters, and stress-testing new RAG evaluation methodologies.