sam-paech/mmlu-pro-nomath-sml

MMLU-Pro-NoMath是MMLU-Pro的一个子集，移除了需要多步计算的数学问题，旨在提供一个更快速运行的评估子集，主要测试知识和推理能力。数据集通过Claude-3.5-sonnet分类器筛选出不需要多步计算的问题，并对问题长度进行了限制，以加快评估速度并减少内存溢出风险。数据集包含测试集和验证集，分别有2639和70个样本。

The MMLU-Pro-NoMath dataset is a subset of MMLU-Pro, specifically designed to exclude questions requiring multi-step calculations (43% of the original test set). It includes features such as question_id, question, options, answer, answer_index, cot_content, category, and src. Divided into test and validation splits with 2639 and 70 examples respectively, this subset aims to provide a quick-to-run version of MMLU-Pro that is friendly to logprobs evaluations, primarily assessing knowledge and reasoning capabilities. By using the Claude-3.5-sonnet classifier to identify and remove questions requiring multi-step calculations, and by constraining question lengths to optimize evaluation efficiency, this subset retains the essence of MMLU-Pro while enhancing its usability for certain types of evaluations.