MMOU

MMOU（Massive Multi-Task Omni Understanding and Reasoning Benchmark）是一个用于评估多模态模型在长且复杂的真实世界视频中联合理解与推理能力的基准数据集。该数据集的核心挑战在于要求模型在长时间跨度内进行紧密耦合的音频-视觉理解，而非单一模态的孤立识别。 MMOU包含15,000个问答对，源自9,038个网络收集的视频，平均视频时长为711.6秒，覆盖10个主要领域（如学术讲座、动画、日常生活等）和36个子类别。每个问题标注了13种推理技能中的一种或多种，平均每个问题需要同时运用约3种技能。 数据集的构建过程包括专家问题编写、时间戳标注、多选题转换（生成9个干扰项）和质量控制。评估方法包括多选题评估（10个选项）和开放式评估，采用微平均准确率作为主要指标。 初步评估显示，人类表现（84.3%）远超当前最强模型（如Gemini 2.5 Pro的64.2%），凸显了该基准的挑战性。数据集特别强调跨模态推理和长视频理解能力，旨在推动多模态模型在真实场景中的应用。MMOU (Massive Multi-Task Omni Understanding and Reasoning Benchmark) is a benchmark dataset for evaluating the joint understanding and reasoning capabilities of multimodal models in long and complex real-world videos. The core challenge of this dataset requires models to perform tightly coupled audio-visual understanding over long time spans, rather than isolated recognition of single modalities. MMOU contains 15,000 question-answer pairs sourced from 9,038 web-collected videos, with an average video duration of 711.6 seconds. It covers 10 major domains (such as academic lectures, animations, daily life, etc.) and 36 subcategories. Each question is annotated with one or more of 13 reasoning skills, with an average of approximately 3 skills required per question. The dataset construction process includes expert-written question compilation, timestamp annotation, multiple-choice question conversion (generating 9 distractors), and quality control. Evaluation methods include 10-option multiple-choice assessment and open-ended evaluation, with micro-averaged accuracy as the primary evaluation metric. Preliminary evaluation results show that human performance (84.3%) far exceeds that of current state-of-the-art models (e.g., Gemini 2.5 Pro with 64.2%), highlighting the high difficulty of this benchmark. The dataset specifically focuses on cross-modal reasoning and long-video understanding capabilities, aiming to promote the application of multimodal models in real-world scenarios.