daman1209arora/jeebench

JEEBench是一个用于评估大型语言模型（LLMs）在解决复杂问题上的能力的基准数据集。该数据集包含515个来自IIT JEE-Advanced考试的数学、物理和化学难题，要求模型具备深度的领域知识和长程推理能力。评估结果表明，即使使用了自一致性、自优化和思维链提示等技术，最高性能仍低于40%。GPT-4作为最佳模型，其典型失败模式包括代数操作错误、难以将抽象概念准确转化为数学方程以及未能检索到相关领域特定概念。此外，GPT-4无法通过提示评估因错误答案引入的负分风险，为此开发了一种基于自一致性的后验置信度阈值方法，以实现有效的响应选择。

原始信息汇总

JEEBench 数据集概述

基本信息

• 许可证: MIT
• 任务类别: 问答
• 语言: 英语
• 标签: 化学, 物理, 数学
• 名称: JEEBench
• 数据集大小: 小于1K

详细描述

JEEBench 是一个用于评估大型语言模型问题解决能力的挑战性基准数据集。该数据集包含从竞争激烈的 IIT JEE-Advanced 考试中精选的 515 道难题，涵盖数学、物理和化学领域。解决这些问题需要深入的领域知识和长程推理能力。

引用信息

如果您在研究中使用该数据集，请使用以下引用格式： latex @inproceedings{arora-etal-2023-llms, title = "Have {LLM}s Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models", author = "Arora, Daman and Singh, Himanshu and {Mausam}", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.emnlp-main.468", doi = "10.18653/v1/2023.emnlp-main.468", pages = "7527--7543", abstract = "The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past years. In response, we present JEEBench, a considerably more challenging benchmark dataset for evaluating the problem solving abilities of LLMs. We curate 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deep in-domain knowledge is essential for solving problems in this benchmark. Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40{%}. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. We also observe that by mere prompting, GPT-4 is unable to assess risk introduced by negative marking for incorrect answers. For this, we develop a post-hoc confidence-thresholding method over self-consistency, which enables effective response selection. We hope that our challenging benchmark will guide future re-search in problem-solving using LLMs.", }