AQA-Bench

AQA-Bench是一个专为评估大型语言模型（LLMs）在算法上下文中的顺序推理能力而设计的新基准。该数据集由加州大学圣克鲁兹分校和爱丁堡大学的研究团队创建，包含12个不同的LLMs评估。数据集的核心特点在于其交互式评估协议，例如在深度优先搜索（DFS）中，每个节点的连接边是否可用取决于模型对该节点的遍历，从而要求LLM有效地记忆已访问节点并策略化后续移动。AQA-Bench构建了三种不同的算法环境：二分搜索、深度优先搜索和广度优先搜索，以评估LLMs的顺序推理能力。该数据集旨在解决LLMs在执行复杂、顺序推理任务时的能力评估问题，并希望推动未来在理解和增强LLMs顺序推理能力方面的工作。AQA-Bench is a novel benchmark specifically designed to evaluate the sequential reasoning capabilities of large language models (LLMs) in algorithmic contexts. Developed by research teams from the University of California, Santa Cruz and the University of Edinburgh, this dataset includes evaluations across 12 distinct LLMs. The core highlight of this dataset lies in its interactive evaluation protocol. For instance, in depth-first search (DFS), the availability of each node's connecting edges depends on the model's traversal of that node, which requires LLMs to effectively memorize visited nodes and strategize subsequent moves. AQA-Bench constructs three distinct algorithmic environments: binary search, depth-first search, and breadth-first search, to assess the sequential reasoning abilities of LLMs. This benchmark aims to address the gap in evaluating LLMs' performance on complex sequential reasoning tasks, and seeks to advance future work focused on understanding and enhancing the sequential reasoning capabilities of LLMs.