Benchmarking Reasoning Beyond Correctness using Structural Consistency

This is the WMPS (Word Math Problems) dataset which includes modifications to existing benchmarks to create additional variants of each problem.

Current AI benchmarks often overestimate model reasoning capabilities with unreliable measures that prioritize answer correctness over the reasoning process itself. Consequently, AI systems can achieve high scores by employing ‘shortcut’ strategies—such as memorization, superficial pattern matching, or heuristic guessing—rather than demonstrating sound inference. While it would be ideal to evaluate the process or justification for the answer, there are immense practical difficulties in generating checkable proofs, evaluating the inference-time traces, or mechanistic interpretability. We introduce structural consistency, a novel methodology for designing more robust reasoning benchmarks (or for augmenting existing benchmarks). This approach involves systematically generating multiple, structurally distinct yet logically (or semantically) related variants for each original task instance. The central premise is that a system consistently and correctly solving all variants is more likely to be employing a robust reasoning process, rather than relying on superficial heuristics. Our experiments demonstrate that, while all evaluated models exhibit a performance decline when faced with these variant sets, their ability to maintain structural consistency varies significantly. This reveals critical differences in their reasoning robustness that are often obscured by standard benchmark evaluations, thereby underscoring the utility of our approach for a more rigorous assessment of AI reasoning capabilities.