aashiqmuhamed/RefusalBench-GaRAGe
收藏资源简介：
RefusalBench-GaRAGe是一个多文档诊断基准，用于评估检索增强生成（RAG）中的选择性拒绝能力：即基于基础的语言模型在检索到的段落支持可靠答案时回答，否则以正确理由拒绝。该基准是论文《RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models》中发布的两个基准之一。每个实例对已验证可回答的GaRAGe问题应用了176种语言扰动杠杆（6个不确定性类别 × 3种强度），其上下文被标准化为10个段落（信号+噪声），然后通过一个要求一致同意的4模型生成-验证管道进行过滤。数据集包含1,506个实例（单个测试分割），基于91个源GaRAGe问题，涵盖5个领域。任务要求模型根据查询和10个基础段落输出答案（引用段落）或单个拒绝代码，地面真实标签为expected_rag_behavior。拒绝代码包括REFUSE_AMBIGUOUS_QUERY等，扰动类别包括P-Ambiguity等。数据分布自然不平衡，反映了每种扰动类型通过一致验证的频率。
RefusalBench-GaRAGe is a multi-document diagnostic benchmark for evaluating the selective refusal capability of retrieval-augmented generation (RAG): i.e., grounded language models will answer when retrieved passages support a reliable answer, and refuse to answer with correct justifications otherwise. This benchmark is one of the two benchmarks released in the paper *RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models*. Each instance applies 176 language perturbation levers (6 uncertainty categories × 3 intensity levels) to a verified answerable GaRAGe question, whose context is standardized to 10 passages (signal + noise), before being filtered via a 4-model generation-verification pipeline that mandates unanimous agreement. The dataset comprises 1,506 instances (single test split), built upon 91 source GaRAGe questions across 5 domains. The task requires models to output either an answer (with cited passages) or a single refusal code based on the given query and 10 grounding passages, with the ground-truth label being "expected_rag_behavior". Refusal codes include REFUSE_AMBIGUOUS_QUERY and other similar codes, while perturbation categories include P-Ambiguity and other similar categories. The data distribution is naturally imbalanced, which reflects the frequency with which each perturbation type passes the unanimous verification.
