MURU-BENCH: A Benchmark for Mathematical Reasoning Under Uncertainty

MURU-BENCH is a 3,000-problem benchmark for evaluating mathematical reasoning under uncertainty in large language models. Unlike existing benchmarks (GSM8K, MATH, MMLU) that score correctness as a single number, MURU-BENCH requires every prediction to include a point estimate, a calibrated confidence interval, a stated confidence probability, and an identified reasoning framework. Problems span five categories:- Bayesian Updating (910 problems)- Distribution Estimation (660 problems)- Decision Under Uncertainty (525 problems)- Adversarial Ambiguity (474 problems)- Conditional Probability Chains (431 problems) Five difficulty levels (D1-D5) are generated by 21 parametric templates with closed-form solutions, guaranteeing correctness by construction. The evaluation harness reports six metrics including Expected Calibration Error and a safety-relevant overconfidence rate, and supports six API providers (OpenAI, Anthropic, Google, Groq, OpenRouter, Together AI). GitHub: https://github.com/swetank18/MURU_proj