yj-zhao/PaperMind

PaperMind是一个用于评估多模态大语言模型（LLMs）在科学论文上的代理推理和批判能力的基准测试数据集。它包含四个互补的任务：Q1（在科学背景下对视觉内容进行精确描述）、Q2（解释实验结果并生成连贯的分析叙述）、Q3（使用工具进行代理推理以从多篇论文或外部来源检索和综合证据）和Q4（基于真实同行评审场景，批判性地检查科学主张，澄清模糊之处，识别弱点、缺失证据或不明确的假设）。数据集涵盖了农业、生物学、化学、计算机科学、医学、物理学和经济学等多个科学领域。

PaperMind is a benchmark for evaluating agentic reasoning and critique over scientific papers in multimodal LLMs. It evaluates LLMs on four complementary tasks built from scientific papers: Q1 (grounding visual content in scientific context and producing precise descriptions), Q2 (interpreting experimental results and generating coherent analytical narratives), Q3 (performing agentic reasoning with tool use to retrieve and synthesize evidence from multiple papers or external sources), and Q4 (critically examining scientific claims by clarifying ambiguities, identifying weaknesses, missing evidence, or unclear assumptions based on real peer-review scenarios). Supported domains of scientific papers include agriculture, biology, chemistry, cs, medicine, physics, and economics.