SimpleQA-Verified

SimpleQA Verified is a 1,000-prompt benchmark for reliably evaluating Large Language Models (LLMs) on short-form factuality and parametric knowledge. The authors from Google DeepMind and Google Research address various limitations of SimpleQA, originally designed by Wei et al. (2024) at OpenAI, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created to provide the research community with a more precise instrument to track genuine progress in factuality, discourage overfitting to benchmark artifacts, and ultimately foster the development of more trustworthy AI systems.

Lukas Haas and Gal Yona and Giovanni D'Antonio and Sasha Goldshtein and Dipanjan Das. SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge. https://arxiv.org/abs/2509.07968, 2025. Google DeepMind, Google Research.

SimpleQA Verified 是一款含千条提示词的基准评测集，用于精准评估大语言模型（Large Language Models, LLMs）的短格式事实性能力与参数化知识储备。来自谷歌深度思维（Google DeepMind）与谷歌研究院（Google Research）的作者团队，针对OpenAI Wei团队于2024年最初提出的原版SimpleQA所存在的多重不足进行了优化，这些不足包括标签存在噪声与错误、主题偏见以及问题冗余现象。SimpleQA Verified 的研发旨在为学界提供更为精准的评测工具，用以追踪事实性任务领域的真实研究进展，防止模型过拟合基准测试的人工构造特征，并最终助力更具可信性的人工智能系统的研发。

Lukas Haas、Gal Yona、Giovanni D'Antonio、Sasha Goldshtein与Dipanjan Das. SimpleQA Verified：一款用于评测参数化知识的可靠事实性基准数据集. https://arxiv.org/abs/2509.07968, 2025. Google DeepMind, Google Research.

搜集汇总

背景与挑战

背景概述

SimpleQA Verified是一个包含1000个提示的基准数据集，旨在可靠评估大语言模型在短篇事实性和参数知识方面的表现。它解决了原始SimpleQA中的标签噪声、主题偏见和冗余问题，为研究社区提供了更精确的工具，以追踪事实性进展并促进可信赖AI系统的发展。