AxiomicLabs/ArithMark

ArithMark是一个用于评估语言模型算术推理能力的程序生成基准测试数据集。每个问题呈现一个数值表达式，要求模型从四个选项（包括整数和浮点数）中识别正确结果。与基于知识的基准测试不同，该数据集不包含模型可能从预训练中记忆的事实，所有问题均通过随机整数和运算符序列生成，确保模型无法通过模式匹配训练数据，必须实际进行计算。这使其成为直接探测模型数值推理能力的工具，独立于世界知识或表面启发式方法。评估采用对数似然多项选择方法，无需思维链或提示技巧。数据集包含5个难度阶段，按操作数数量递增，每个阶段有1000个问题，总计5000个问题。数据格式为JSON，包括ID、阶段、操作数数量、表达式、选项、答案索引和答案等字段。设计决策包括从左到右评估表达式（无运算符优先级）、整数安全结果（浮点数四舍五入到4位小数）、操作数范围（整数从[-20, 20]采样）以及干扰项生成机制（确保答案类型不可唯一识别）。ArithMark is a programmatically generated benchmark dataset for evaluating the arithmetic reasoning capabilities of language models. Each question presents a numerical expression, requiring the model to identify the correct result from four options including both integers and floating-point numbers. Unlike knowledge-based benchmarks, this dataset contains no facts that models might memorize during pre-training. All questions are generated from random integer and operator sequences, ensuring that models cannot rely on pattern-matching training data and must perform actual computations. This makes it a tool for directly probing the numerical reasoning capabilities of models, independent of world knowledge or surface-level heuristics. Evaluation adopts the log-likelihood multiple-choice method, without requiring Chain-of-Thought or prompt engineering techniques. The dataset includes 5 difficulty stages, with the number of operands increasing progressively. Each stage contains 1000 questions, totaling 5000 questions overall. The data is formatted in JSON, with fields including ID, stage, number of operands, expression, options, answer index, and answer. Design decisions include left-to-right evaluation of expressions without operator precedence, safe handling of integer results, rounding floating-point values to four decimal places, sampling integers within the range [-20, 20] for operands, and a distractor generation mechanism to ensure that the answer type cannot be uniquely identified.