Randolphzeng/Mr-Ben

该数据集名为Meta-Reasoning Benchmark，旨在通过元推理范式全面评估大型语言模型（LLMs）的推理能力。每个数据点包含三个关键元素：问题、CoT答案和错误分析。评估模型需要判断解决方案的正确性，并报告第一个错误步骤和错误原因。数据集中的每个问题都有三个候选解决方案，分别来自Mistral、Claude和GPT3.5模型。数据格式包括问题的唯一标识符、所属主题、问题内容、选项、真实分析、真实答案、修改后的真实答案、采样模型、模型解决方案步骤、模型解决方案的正确性、第一个错误步骤、错误原因和纠正后的第一个错误步骤。

Mr-Ben is a comprehensive meta-reasoning benchmark designed to evaluate the reasoning capabilities of large language models. This benchmark employs a meta-reasoning paradigm, casting LLMs in the role of a teacher to assess the correctness of the reasoning process, analyze errors, and provide corrections. Each data point consists of a question, a CoT answer, and an error analysis. The dataset includes three sampled solutions from Mistral, Claude, and GPT3.5 for each question, with each question having a unique identifier, subject, question, options, ground truth analysis, ground truth answer, sampled model, model solution steps, solution correctness, first error step of the solution, error reason, and rectified first error step. The evaluation results are scored using the MR-Score, which consists of the Matthews Correlation Coefficient, the ratio of solutions with correctly predicted first error steps, and the ratio of solutions with correctly predicted first error steps plus error reasons.