ServiceNow-AI/Abstain-QA

Abstain-QA是一个评估大型语言模型拒绝回答能力的数据集，包含2900个多选问题样本，涵盖从简单的事实性问题到复杂的逻辑和概念推理挑战。样本来源于Pop-QA、MMLU和专门针对卡纳提克音乐知识盲区设计的CQA数据集。每个样本都包含一个明确的“我不知道/以上都不是”选项，用于衡量LLM的拒绝回答。

Abstain-QA is a dataset designed to evaluate the Abstention Ability of Large Language Models (LLMs), consisting of 2900 multiple-choice question answering (MCQA) samples. It covers a range from straightforward factual inquiries to complex logical and conceptual reasoning challenges. The samples are sourced from Pop-QA, MMLU, and the CQA dataset, specifically created for this work to address the gap in under-represented knowledge domains related to Carnatic Music. Each sample includes an explicit I Dont Know/None of the above option to measure LLMs abstentions.