MM-NIAH

该数据集是一项专为系统评估现有多模态大型语言模型（MLLMs）理解长篇多模态文档能力而设计的基准测试。该基准测试对领先的多模态语言模型进行了评估，并将人类表现作为基准线。它着重考察模型处理长篇多模态文档的能力，并突显了随着上下文长度增加，性能下降的问题。测试任务包括多模态检索、计数和推理。

This dataset is a benchmark specifically designed for the systematic evaluation of existing Multimodal Large Language Models (MLLMs) in their ability to comprehend long-form multimodal documents. This benchmark evaluates state-of-the-art multimodal language models, with human performance serving as the baseline. It focuses on examining models' capabilities to process long-form multimodal documents, and highlights the performance degradation issue that occurs as context length increases. The test tasks include multimodal retrieval, counting, and reasoning.