ewdfd/SMART

--- pretty_name: SMART language: - en license: mit task_categories: - question-answering - text-generation - text-classification tags: - mathematics - reasoning - llm-evaluation - benchmark - education - chain-of-thought size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: test path: SMART.jsonl --- # SMART: Evaluating LLMs’ Mathematical Reasoning via a Human Cognitive Process-Inspired Benchmark SMART is a fine-grained benchmark for evaluating large language models (LLMs) on mathematical reasoning from a human cognitive process perspective. Instead of evaluating only the final answer, SMART decomposes mathematical problem solving into four cognitive dimensions inspired by Pólya’s problem-solving theory: 1. **Semantic Understanding** 2. **Mathematical Reasoning** 3. **Arithmetic Computation** 4. **Reflection & Refinement** SMART is designed to diagnose where a model succeeds or fails during the problem-solving process, rather than reducing reasoning to a shallow input-output mapping. The benchmark contains **10,000 test instances**, including **2,000 seed questions** and **8,000 dimension-specific task variants**. ## Data Fields Each SMART instance contains the following fields: - `question`: the original seed math word problem. - `notation`: the notation-based arithmetic form derived from the original problem, used to isolate arithmetic computation. - `background`: the structured background information extracted from the question, including the problem scenario, goal, known and unknown quantities, relationships and constraints, and potentially irrelevant information. - `smt-lib`: the SMT-LIB symbolic formalization of the problem, used to evaluate mathematical reasoning through executable logical structure. - `answer`: the answer associated with the instance. - `gt_answer`: the ground-truth final answer. - `gt_cot`: the ground-truth chain-of-thought solution. - `w_cot`: a corrupted or incorrect chain-of-thought solution with injected errors, used for Reflection & Refinement evaluation. - `wrong`: an indicator of whether the `w_cot` contains an injected error pattern. Notes: - `question` corresponds to the original mathematical problem. - `background` is mainly used for the Understanding dimension. - `smt-lib` is mainly used for the Reasoning dimension. - `notation` is mainly used for the Arithmetic dimension. - `gt_cot`, `w_cot`, and `wrong` are mainly used for the Reflection & Refinement dimension. An example structure is shown below: ```json { "question": "Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?", "notation": "a=80000, b=50000, c=a+b, d=1.5a, e=d+a, f=e-c, f?", "background": "```json\n{\n \"problem_description\": {\n \"problem_scenario\": \"Josh buys a house, invests in repairs, and sells it for profit.\",\n \"goal\": \"Calculate the profit Josh made from flipping the house.\"\n },\n \"quantities\": {\n \"known\": [\n \"Initial house purchase cost: $80,000\",\n \"Cost of repairs: $50,000\",\n \"Increase in house value: 150%\"\n ],\n \"unknown\": [\n \"Profit made from flipping the house\"\n ]\n },\n \"relationships_and_constraints\": [\n \"Total cost = purchase cost + repair cost\",\n \"House value increases by a specified percentage\",\n \"Profit = selling price - total cost\"\n ],\n \"potentially_irrelevant_info\": []\n}\n```", "smt-lib": " (set-logic QF_NRA)(declare-fun a () Real) (declare-fun b () Real) (declare-fun c () Real) (declare-fun d () Real) (declare-fun e () Real) (declare-fun f () Real) (assert (= a 80000))(assert (= b 50000))(assert (= c (+ a b)))(assert (= d (* a 1.5)))(assert (= e (+ d a)))(assert (= f (- e c)))(check-sat)(get-value (f))", "answer": 70000.0, "gt_answer": 70000.0, "gt_cot": "The cost of the house and repairs came out to 80,000+50,000=$<<80000+50000=130000>>130,000\nHe increased the value of the house by 80,000*1.5=<<80000*1.5=120000>>120,000\nSo the new value of the house is 120,000+80,000=$<<120000+80000=200000>>200,000\nSo he made a profit of 200,000-130,000=$<<200000-130000=70000>>70,000\n#### 70000", "w_cot": "The cost of the house and repairs came out to 80,000+50,000=$<<80000+50000=130000>>130,000.He increased the value of the house by 80,000*1.5=<<80000*1.5=144561>>120,000.So the new value of the house is 120,000+80,000=$<<120000+80000=200000>>200,000.So he made a profit of 200,000-130,000=$<<200000-130000=70000>>70,000.#### 70000.", "wrong": 1 }--- ### 数据集元数据 - 数据集名称：SMART - 语言：英语 - 许可证：MIT - 任务类别：问答、文本生成、文本分类 - 标签：数学、推理、大语言模型评估、基准测试、教育、思维链（chain-of-thought） - 样本规模类别：10000 < 样本量 < 100000 - 配置：默认配置，数据文件为测试集的SMART.jsonl --- # SMART：基于人类认知过程启发的基准测试，用于评估大语言模型（Large Language Model，LLM）的数学推理能力 SMART是一款细粒度基准测试集，用于从人类认知过程的视角评估大语言模型的数学推理能力。与仅评估最终答案的传统方式不同，SMART借鉴波利亚（Pólya）的问题求解理论，将数学解题流程拆解为四大认知维度： 1. **语义理解** 2. **数学推理** 3. **算术计算** 4. **反思与优化** SMART的设计初衷是诊断模型在解题全流程中的优劣环节，而非将推理简化为浅层的输入-输出映射。该基准测试集共包含**10000个测试样本**，其中涵盖**2000个种子问题**与**8000个维度专属的任务变体**。 ## 数据字段 每个SMART样本均包含以下字段： - `question`：原始种子数学应用题。 - `notation`：从原始问题推导得到的基于符号的算术表达式，用于隔离算术计算环节。 - `background`：从问题中提取的结构化背景信息，涵盖问题场景、解题目标、已知与未知量、关系约束及潜在无关信息。 - `smt-lib`：问题的SMT-LIB符号化形式，用于通过可执行逻辑结构评估数学推理能力。 - `answer`：与该样本关联的答案。 - `gt_answer`：真实最终标准答案（ground-truth answer）。 - `gt_cot`：真实思维链（chain-of-thought）解题过程。 - `w_cot`：带有注入错误的损坏或错误思维链解题过程，用于反思与优化维度的评估。 - `wrong`：用于标记`w_cot`是否包含注入错误模式的指标。 ## 备注 - `question` 对应原始数学问题。 - `background` 主要用于语义理解维度。 - `smt-lib` 主要用于数学推理维度。 - `notation` 主要用于算术计算维度。 - `gt_cot`、`w_cot` 与 `wrong` 主要用于反思与优化维度。 以下为示例结构： json { "question": "Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?", "notation": "a=80000, b=50000, c=a+b, d=1.5a, e=d+a, f=e-c, f?", "background": "json { "problem_description": { "problem_scenario": "Josh buys a house, invests in repairs, and sells it for profit.", "goal": "Calculate the profit Josh made from flipping the house." }, "quantities": { "known": [ "Initial house purchase cost: $80,000", "Cost of repairs: $50,000", "Increase in house value: 150%" ], "unknown": [ "Profit made from flipping the house" ] }, "relationships_and_constraints": [ "Total cost = purchase cost + repair cost", "House value increases by a specified percentage", "Profit = selling price - total cost" ], "potentially_irrelevant_info": [] } ", "smt-lib": " (set-logic QF_NRA)(declare-fun a () Real) (declare-fun b () Real) (declare-fun c () Real) (declare-fun d () Real) (declare-fun e () Real) (declare-fun f () Real) (assert (= a 80000))(assert (= b 50000))(assert (= c (+ a b)))(assert (= d (* a 1.5)))(assert (= e (+ d a)))(assert (= f (- e c)))(check-sat)(get-value (f))", "answer": 70000.0, "gt_answer": 70000.0, "gt_cot": "The cost of the house and repairs came out to 80,000+50,000=$<<80000+50000=130000>>130,000 He increased the value of the house by 80,000*1.5=<<80000*1.5=120000>>120,000 So the new value of the house is 120,000+80,000=$<<120000+80000=200000>>200,000 So he made a profit of 200,000-130,000=$<<200000-130000=70000>>70,000 #### 70000", "w_cot": "The cost of the house and repairs came out to 80,000+50,000=$<<80000+50000=130000>>130,000.He increased the value of the house by 80,000*1.5=<<80000*1.5=144561>>120,000.So the new value of the house is 120,000+80,000=$<<120000+80000=200000>>200,000.So he made a profit of 200,000-130,000=$<<200000-130000=70000>>70,000.#### 70000.", "wrong": 1 }