MBZUAI/TABVERSE
收藏资源简介：
TABVERSE是一个受控的多模态表格理解基准数据集，旨在对齐HTML、Markdown和LaTeX三种表格表示形式与渲染的PNG图像，从而系统评估大型语言模型（LLMs）和视觉语言模型（VLMs）在跨格式和模态交互中的表现。数据集包含两个配置：qa配置用于任务预测（问答），包含700个问题-表格对（350个简单和350个困难），每个对提供自然语言问题、黄金答案、难度等级和类别，以及表格在三种格式下的完整渲染图像和源代码；suc配置用于结构化理解与理解（SUC）和格式生成（结构重建），包含629个唯一表格，每个表格提供结构探测任务的黄金答案，并支持六种格式转换方向（如HTML↔Markdown）。数据集来源于多个公开数据集（如FEVEROUS、HybridQA、SQA、TabFact、ToTTo），并经过清洗以确保表格内容在三种视图中保持一致。任务包括结构化理解、问答和格式生成，评估了17种模型（包括开源和闭源LLMs和VLMs）在三种输入模式（纯文本、图像、VLM文本）下的性能，主要使用精确匹配准确率作为评估指标。该数据集旨在推动表格理解研究，特别是在多模态和跨格式场景下的模型能力测试。
TABVERSE is a controlled multimodal table benchmark that aligns HTML, Markdown, and LaTeX table representations with rendered PNG images, enabling systematic evaluation of how format and modality interact in large language models (LLMs) and vision-language models (VLMs), with table content held fixed across all three views. The dataset includes two configurations: the qa config for task prediction (question answering) with 700 question-table pairs (350 Easy and 350 Hard), each providing a natural-language question, gold label, difficulty, category, and the full table rendered in all three formats as images and source code; and the suc config for Structured Understanding & Comprehension (SUC) and format generation (Structure Reconstruction) with 629 unique tables, each supplying ground-truth answers for structural probing tasks and supporting six conversion directions (e.g., HTML↔Markdown). It is sourced from multiple public datasets (e.g., FEVEROUS, HybridQA, SQA, TabFact, ToTTo) and cleaned to ensure consistency across views. Tasks include structured understanding, QA, and format generation, evaluating 17 models (including open-weight and closed LLMs and VLMs) in three pipeline modes (LLM text, VLM image, VLM text), with exact-match accuracy as the primary metric. The dataset aims to advance table understanding research, particularly in multimodal and cross-format scenarios.
