MMLU (Massive Multitask Language Understanding)

MMLU (大规模多任务语言理解) 是一种新的基准，旨在通过仅在零射击和少射击设置中评估模型来衡量预训练期间获得的知识。这使得基准测试更具挑战性，更类似于我们评估人类的方式。该基准涵盖了STEM，人文学科，社会科学等领域的57个主题。它的难度从初级水平到高级专业水平，它考验世界知识和解决问题的能力。学科范围从传统领域 (例如数学和历史) 到更专业的领域 (例如法律和道德)。对象的粒度和广度使基准成为识别模型盲点的理想选择。

MMLU (Massive Multitask Language Understanding) is a novel benchmark designed to measure knowledge acquired during model pre-training by exclusively evaluating models in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate human beings. The benchmark covers 57 topics across STEM, humanities, social sciences, and other fields. Its difficulty ranges from introductory level to advanced professional level, and it tests both world knowledge and problem-solving capabilities. The subject scope spans from traditional fields such as mathematics and history to more specialized domains including law and ethics. The granularity and breadth of this benchmark make it an ideal choice for identifying model blind spots.

搜集汇总

背景与挑战

背景概述

MMLU是一个大规模多任务语言理解基准，用于在零射击和少射击设置下评估预训练模型的知识掌握程度。它覆盖57个学科，难度从初级到专业，旨在测试模型的世界知识和问题解决能力。