MURU-BENCH: A Benchmark for Mathematical Reasoning Under Uncertainty

MURU-BENCH is a 3,000-problem benchmark for evaluating mathematical reasoning under uncertainty in large language models. Unlike existing benchmarks (GSM8K, MATH, MMLU) that score correctness as a single number, MURU-BENCH requires every prediction to include a point estimate, a calibrated confidence interval, a stated confidence probability, and an identified reasoning framework.

Problems span five categories:- Bayesian Updating (910 problems)- Distribution Estimation (660 problems)- Decision Under Uncertainty (525 problems)- Adversarial Ambiguity (474 problems)- Conditional Probability Chains (431 problems)

Five difficulty levels (D1-D5) are generated by 21 parametric templates with closed-form solutions, guaranteeing correctness by construction. The evaluation harness reports six metrics including Expected Calibration Error and a safety-relevant overconfidence rate, and supports six API providers (OpenAI, Anthropic, Google, Groq, OpenRouter, Together AI).

GitHub: https://github.com/swetank18/MURU_proj

MURU-BENCH 是一款包含3000道试题的基准测试集，用于评估大语言模型（Large Language Model，LLM）在不确定性场景下的数学推理能力。与现有基准（GSM8K、MATH、MMLU）仅以单一数值衡量正确性不同，MURU-BENCH要求每一条预测结果都需包含点估计、校准后的置信区间、明确的置信概率，以及经标识的推理框架。

试题涵盖五大类别：
- 贝叶斯更新（Bayesian Updating，910道题）
- 分布估计（Distribution Estimation，660道题）
- 不确定性决策（Decision Under Uncertainty，525道题）
- 对抗歧义（Adversarial Ambiguity，474道题）
- 条件概率链（Conditional Probability Chains，431道题）

该基准通过21个带有闭式解的参数化模板生成了D1至D5共五个难度层级，从构建层面确保了试题的正确性。其评测套件可输出六项评估指标，其中包括预期校准误差（Expected Calibration Error）以及与安全性相关的过度自信率，同时支持六大API提供商：OpenAI、Anthropic、Google、Groq、OpenRouter、Together AI。

GitHub 仓库地址：https://github.com/swetank18/MURU_proj