toloka/u-math

U-MATH是一个包含1100个未发表的大学水平数学问题的综合基准数据集，这些问题来源于真实的教学材料。该数据集旨在评估大型语言模型（LLMs）的数学推理能力。数据集涵盖了六个核心数学主题，并包含20%的多模态问题（涉及视觉元素如图表和图形）。数据集的特征包括问题ID、主题标签、是否包含图像、图像数据、问题陈述和正确答案。数据集的使用方法、评估指标和引用信息也提供了详细说明。

U-MATH is a comprehensive benchmark dataset of 1,100 unpublished university-level math problems sourced from real teaching materials. The dataset is designed to evaluate the mathematical reasoning capabilities of Large Language Models (LLMs). It covers six core mathematical topics in a balanced manner and includes 20% of multimodal problems (involving visual elements such as graphs and diagrams). The dataset features include problem ID, subject tag, whether the problem includes an image, image data, problem statement, and golden answer. Evaluation metrics include accuracy, with splits by subject and problem type (text-only vs multimodal).