ArtificialAnalysis/AA-Omniscience-Public

AA-Omniscience-Public 是一个包含600个问题的数据集，涵盖多个领域，用于测试模型的知识和幻觉倾向。该数据集是完整6,000个问题数据集的10%子集，确保在发布时能够代表完整数据集的性能。问题由自动生成代理创建，经过过滤和修订，确保问题难度高、无歧义、不依赖特定来源且答案精确。数据集旨在衡量模型准确回忆事实信息的能力，并在知识不足时正确避免回答。评分方法使用分级模型对答案进行分类，确保评估的准确性。AA-Omniscience-Public contains 600 questions across a wide range of domains used to test a model’s knowledge and hallucination tendencies. It is a 10% subset of the full 6,000-question dataset, sampled to ensure representativeness of model performance at the time of release. Questions are created using an automated question generation agent, filtered and revised to be difficult, unambiguous, not reliant on specific sources, and precise. The dataset is designed to measure a model’s ability to recall factual information accurately across domains and correctly abstain when its knowledge is insufficient. A grading model is used to classify answers, ensuring evaluation integrity.

搜集汇总

背景与挑战

背景概述

AA-Omniscience-Public是一个包含600个问答对的数据集，旨在评估大语言模型在多个经济相关领域（如金融、会计、商业管理）中的知识准确性和幻觉倾向。该数据集通过自动化问题生成代理创建，问题设计为难度高、答案精确且不依赖特定来源，并采用特殊评分机制鼓励模型在不确定时避免回答，以衡量其知识可靠性。