chenweijie/OmniDocBench

# OmniDocBench [English](./README.md) | [简体中文](./README_ZH.md) **OmniDocBench** is an evaluation dataset for diverse document parsing in real-world scenarios, with the following characteristics: - **Diverse Document Types**: The evaluation set contains 1355 PDF pages, covering 9 document types, 4 layout types and 3 language types. It has broad coverage including academic papers, financial reports, newspapers, textbooks, handwritten notes, etc. - **Rich Annotations**: Contains location information for 15 block-level (text paragraphs, titles, tables, etc., over 20k in total) and 4 span-level (text lines, inline formulas, superscripts/subscripts, etc., over 80k in total) document elements, as well as recognition results for each element region (text annotations, LaTeX formula annotations, tables with both LaTeX and HTML annotations). OmniDocBench also provides reading order annotations for document components. Additionally, it includes various attribute labels at page and block levels, with 5 page attribute labels, 3 text attribute labels and 6 table attribute labels. - **High Annotation Quality**: Through manual screening, intelligent annotation, manual annotation, full expert quality inspection and large model quality inspection, the data quality is relatively high. - **Evaluation Code Suite**: Designed with end-to-end evaluation and single module evaluation code to ensure fairness and accuracy of evaluation. The evaluation code suite can be found at [OmniDocBench](https://github.com/opendatalab/OmniDocBench). ## Updates - [2025/09/25] Major Update: updated from v1.0 to v1.5: - Images of newspaper and notes have been increased to 200 DPI. - To balance the number of pages in Chinese and English and increase the number of pages containing formulas, 374 new pages have been added, including 25 in Chinese and 349 in English. These pages include books, PPTs, color illustrated textbooks, test papers, magazines, and newspapers. The number of inline formulas has increased from 353 to 1050. - Language attributes have been added to formulas, increasing the number of Chinese formulas to 68 and English formulas to 982. - Fixed typos in some text and table annotations in v1.0. - [2024/12/25] Added PDF format of the evaluation set for models that require PDFs as input for evaluation. Added original PDF slices with metadata. - [2024/12/10] Fixed height and width fields for some samples. This fix only affects page-level height and width fields and does not impact the correctness of other annotations - [2024/12/04] Released OmniDocBench evaluation dataset ## Dataset Introduction The evaluation set contains 1355 PDF pages, covering 9 document types, 4 layout types and 3 language types. OmniDocBench has rich annotations, including 15 block-level annotations (text paragraphs, titles, tables, etc.) and 4 span-level annotations (text lines, inline formulas, superscripts/subscripts, etc.). All text-related annotation boxes contain text recognition annotations, formulas contain LaTeX annotations, and tables contain both LaTeX and HTML annotations. OmniDocBench also provides reading order annotations for document components. Additionally, it includes various attribute labels at page and block levels, with 5 page attribute labels, 3 text attribute labels and 6 table attribute labels.  ## Usage You can use our [evaluation method](https://github.com/opendatalab/OmniDocBench) to conduct evaluations across several dimensions: - End-to-end evaluation: Includes both end2end and md2md evaluation methods - Layout detection - Table recognition - Formula recognition - Text OCR The evaluation dataset files include: - [OmniDocBench.json](OmniDocBench.json) is the annotation file for the evaluation dataset, stored in JSON format. It supports the end2end evaluation method. The structure and fields are explained below. - [images](./images/) are the corresponding evaluation dataset images, for models that require images as input. - [image_to_pdf.py](https://github.com/opendatalab/OmniDocBench/blob/main/tools/image_to_pdf.py) is the script to covert images to PDFs for those models who take only PDFs as input. <details> <summary>Dataset Format</summary> The dataset format is JSON, with the following structure and field explanations: ```json [{ "layout_dets": [ // List of page elements { "category_type": "text_block", // Category name "poly": [ 136.0, // Position information, coordinates for top-left, top-right, bottom-right, bottom-left corners (x,y) 781.0, 340.0, 781.0, 340.0, 806.0, 136.0, 806.0 ], "ignore": false, // Whether to ignore during evaluation "order": 0, // Reading order "anno_id": 0, // Special annotation ID, unique for each layout box "text": "xxx", // Optional field, Text OCR results are written here "latex": "$xxx$", // Optional field, LaTeX for formulas and tables is written here "html": "xxx", // Optional field, HTML for tables is written here "attribute" {"xxx": "xxx"}, // Classification attributes for layout, detailed below "line_with_spans:": [ // Span level annotation boxes { "category_type": "text_span", "poly": [...], "ignore": false, "text": "xxx", "latex": "$xxx$", }, ... ], "merge_list": [ // Only present in annotation boxes with merge relationships, merge logic depends on whether single line break separated paragraphs exist, like list types { "category_type": "text_block", "poly": [...], ... // Same fields as block level annotations "line_with_spans": [...] ... }, ... ] ... ], "page_info": { "page_no": 0, // Page number "height": 1684, // Page height "width": 1200, // Page width "image_path": "xx/xx/", // Annotated page filename "page_attribute": {"xxx": "xxx"} // Page attribute labels }, "extra": { "relation": [ // Related annotations { "source_anno_id": 1, "target_anno_id": 2, "relation": "parent_son" // Relationship label between figure/table and their corresponding caption/footnote categories }, { "source_anno_id": 5, "target_anno_id": 6, "relation_type": "truncated" // Paragraph truncation relationship label due to layout reasons, will be concatenated and evaluated as one paragraph during evaluation }, ] } }, ... ] ``` </details> <details> <summary>Evaluation Categories</summary> Evaluation categories include: ``` # Block level annotation boxes 'title' # Title 'text_block' # Paragraph level plain text 'figure', # Figure type 'figure_caption', # Figure description/title 'figure_footnote', # Figure notes 'table', # Table body 'table_caption', # Table description/title 'table_footnote', # Table notes 'equation_isolated', # Display formula 'equation_caption', # Formula number 'header' # Header 'footer' # Footer 'page_number' # Page number 'page_footnote' # Page notes 'abandon', # Other discarded content (e.g. irrelevant information in middle of page) 'code_txt', # Code block 'code_txt_caption', # Code block description 'reference', # References # Span level annotation boxes 'text_span' # Span level plain text 'equation_ignore', # Formula to be ignored 'equation_inline', # Inline formula 'footnote_mark', # Document superscripts/subscripts ``` </details> <details> <summary>Attribute Labels</summary> Page classification attributes include: ``` 'data_source': #PDF type classification academic_literature # Academic literature PPT2PDF # PPT to PDF book # Black and white books and textbooks colorful_textbook # Colorful textbooks with images exam_paper # Exam papers note # Handwritten notes magazine # Magazines research_report # Research reports and financial reports newspaper # Newspapers 'language': #Language type en # English simplified_chinese # Simplified Chinese en_ch_mixed # English-Chinese mixed 'layout': #Page layout type single_column # Single column double_column # Double column three_column # Three column 1andmore_column # One mixed with multiple columns, common in literature other_layout # Other layouts 'watermark': # Whether contains watermark true false 'fuzzy_scan': # Whether blurry scanned true false 'colorful_backgroud': # Whether contains colorful background, content to be recognized has more than two background colors true false ``` Block level attribute - Table related attributes: ``` 'table_layout': # Table orientation vertical # Vertical table horizontal # Horizontal table 'with_span': # Merged cells False True 'line': # Table borders full_line # Full borders less_line # Partial borders fewer_line # Three-line borders wireless_line # No borders 'language': # Table language table_en # English table table_simplified_chinese # Simplified Chinese table table_en_ch_mixed # English-Chinese mixed table 'include_equation': # Whether table contains formulas False True 'include_backgroud': # Whether table contains background color False True 'table_vertical' # Whether table is rotated 90 or 270 degrees False True ``` Block level attribute - Text paragraph related attributes: ``` 'text_language': # Text language text_en # English text_simplified_chinese # Simplified Chinese text_en_ch_mixed # English-Chinese mixed 'text_background': # Text background color white # Default value, white background single_colored # Single background color other than white multi_colored # Multiple background colors 'text_rotate': # Text rotation classification within paragraphs normal # Default value, horizontal text, no rotation rotate90 # Rotation angle, 90 degrees clockwise rotate180 # 180 degrees clockwise rotate270 # 270 degrees clockwise horizontal # Text is normal but layout is vertical ``` Block level attribute - Formula related attributes: ``` 'formula_type': # Formula type print # Print handwriting # Handwriting 'equation_language' # Formula language equation_en # English equation_ch # Chinese ``` </details> ## Data Display   ## Acknowledgement - Thank [Abaka AI](https://abaka.ai) for supporting the dataset annotation. ## Copyright Statement The PDFs are collected from public online channels and community user contributions. Content that is not allowed for distribution has been removed. The dataset is for research purposes only and not for commercial use. If there are any copyright concerns, please contact [e-mail]. ## Citation ```bibtex @misc{ouyang2024omnidocbenchbenchmarkingdiversepdf, title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations}, author={Linke Ouyang and Yuan Qu and Hongbin Zhou and Jiawei Zhu and Rui Zhang and Qunshu Lin and Bin Wang and Zhiyuan Zhao and Man Jiang and Xiaomeng Zhao and Jin Shi and Fan Wu and Pei Chu and Minghao Liu and Zhenxiang Li and Chao Xu and Bo Zhang and Botian Shi and Zhongying Tu and Conghui He}, year={2024}, eprint={2412.07626}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2412.07626}, } ``` ## Links - Paper: https://huggingface.co/papers/2412.07626 - GitHub: https://github.com/opendatalab/OmniDocBench

搜集汇总

数据集介绍

构建方式

OmniDocBench的构建以真实世界文档解析需求为出发点，通过精心设计的数据采集与标注流程实现。数据集包含1355页PDF，覆盖学术论文、财务报表、报纸、教科书、手写笔记等9种文档类型、4种布局类型及3种语言类型。标注过程采用人工筛选、智能预标注、人工精标、专家全检与大模型质检相结合的多阶段质量控制策略，确保了高精度。每个页面均标注了15种块级元素（如文本段落、标题、表格等，总计超2万个）和4种跨度级元素（如文本行、行内公式、上下标等，总计超8万个）的位置信息，并提供了对应的文本识别、LaTeX公式及表格的LaTeX/HTML识别结果，同时附有阅读顺序标注。此外，还包含5种页面属性标签、3种文本属性标签和6种表格属性标签，构建方式严谨而全面。

特点

OmniDocBench的核心特点在于其多样性与标注丰富性的有机统一。数据集囊括了从学术文献到彩色教科书、从手写笔记到PPT转换PDF的广泛文档类型，并覆盖单栏、双栏、三栏及混合多栏等布局，语言上涵盖英文、简体中文及中英混合。标注深度尤为突出：块级元素提供精确多边形坐标与分类，跨度级元素细粒度捕捉行内公式与上下标；表格同时以LaTeX和HTML格式呈现，公式则提供LaTeX注释。独特的阅读顺序标注与属性标签（如页面水印、模糊扫描、表格边框类型、文本旋转角度等）为多维度评估提供了坚实基础。v1.5版本新增374页以平衡中英文比例并增加公式样本，进一步提升了数据集的代表性与实用性。

使用方法

使用OmniDocBench时，研究者可通过配套的评估代码套件（位于GitHub仓库）进行多维度评测。支持端到端评估（含end2end与md2md两种模式）以及模块化评估，包括布局检测、表格识别、公式识别和文本OCR。数据集以JSON格式提供主注释文件（OmniDocBench.json），内含每个页面的元素列表、位置多边形、忽略标志、阅读顺序、识别结果及属性标签；images文件夹存放对应图像，适用于需要图像输入的模型。对于仅接受PDF输入的模型，提供了图像转PDF的转换脚本。评估过程中，可利用特殊的合并列表处理段落截断关系，并通过关联标注处理图表与标题的父子关系，确保评估的准确性与公平性。

背景与挑战

背景概述

OmniDocBench是由上海人工智能实验室OpenDataLab团队于2024年12月发布的一个面向真实场景的多样化文档解析评估数据集。该数据集由Linke Ouyang、Yuan Qu等研究人员主导构建，旨在解决当前文档解析领域缺乏统一、全面评估基准的困境。随着数字化办公与学术研究的深入，文档解析技术面临从学术论文、财务报表到报纸、手写笔记等复杂版式的挑战，现有数据集往往局限于单一类型或简单布局。OmniDocBench通过涵盖9种文档类型、4种布局类型及3种语言类型，并包含15种块级与4种跨级元素的精细标注，为文档解析研究提供了高标准的评价平台。其发布迅速引起学术界与工业界关注，成为衡量文档解析系统综合性能的重要标杆，推动了该领域从单一任务向多模态、多维度评估的演进。

当前挑战

OmniDocBench所面临的核心挑战首先在于文档解析领域本身的复杂性：真实场景中的文档类型多样，如学术文献、PPT、彩色教材、试卷等，其布局从单栏到多栏、混合栏变化多端，语言涵盖中英文及混合文本，且存在水印、模糊扫描、彩色背景等噪声干扰，导致现有模型在跨类型泛化能力上表现不足。其次，构建过程中需应对精细标注的难题：15种块级与4种跨级元素的定位、识别及阅读顺序标注要求极高精度，尤其是表格的LaTeX与HTML双重标注、公式的语言属性标注，以及段落截断关系等复杂逻辑，均需通过人工筛选、智能标注、专家质检与大型模型质检的多重流程保障质量。此外，数据集版本迭代（如v1.0至v1.5）中需平衡中英文页数、增加含公式页面，并修正早期标注错误，进一步凸显了大规模高质量文档数据集构建的艰巨性。

常用场景

经典使用场景

在文档智能与版面分析领域，OmniDocBench作为一项综合性基准评测数据集，被广泛用于多类型文档的端到端解析任务。其涵盖学术论文、财务报表、报纸、教材、手写笔记等9种文档类型，并融合单栏、双栏、多栏及混合布局，为版面检测、表格识别、公式识别与文本OCR等子任务提供统一评测平台。研究者利用其15种块级与4种跨行级标注，结合阅读顺序与属性标签，可系统评估模型在复杂真实场景下的泛化能力与鲁棒性。

解决学术问题

该数据集有效解决了现有文档解析评测集类型单一、标注粒度粗、缺乏统一评估框架的学术困境。通过提供超过2万个块级元素与8万个跨行级元素的精细标注，以及LaTeX与HTML双格式表格注释，OmniDocBench支持对版面检测、表格结构识别、公式转录与文本识别等子任务的独立或联合评估。其引入的页面属性（如语言、水印、模糊度）与块级属性（如表合并、公式类型），为分析模型在不同干扰因素下的性能退化提供了量化依据，推动了文档解析领域从粗粒度评测向细粒度诊断的范式演进。

衍生相关工作

OmniDocBench的发布催生了多项衍生性研究工作，包括基于多模态大模型的文档解析框架、面向版面元素的细粒度预训练策略，以及融合阅读顺序的图神经网络方法。其提供的评测代码套件（含端到端与单模块评估）被后续研究广泛采纳为基准，例如用于对比不同架构在表格识别与公式转录任务上的性能差异。此外，该数据集的语言与属性标签体系启发了跨语言文档解析评测集的构建，推动了对中英文混合、手写公式等长尾场景的专项研究，形成了以OmniDocBench为锚点的文档智能评测生态。