opendatalab/OHR-Bench

Name: opendatalab/OHR-Bench
Creator: opendatalab
Published: 2025-04-08T18:14:05+08:00

Hugging Face2025-08-28 更新2025-04-08 收录

光学字符识别

自然语言处理

数据链接：

https://hf-mirror.com/datasets/opendatalab/OHR-Bench 数据链接链接失效反馈

官方服务：

资源简介：

--- license: cc-by-4.0 task_categories: - question-answering - visual-document-retrieval tags: - ocr - rag - retrieval-augmented-generation - benchmark - document-understanding - document-analysis configs: - config_name: default data_files: - split: train path: OHR-Bench_v2.parquet --- This repository contains the OHR-Bench dataset and evaluation framework from the paper [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://huggingface.co/papers/2412.02592). **[📚 Paper](https://huggingface.co/papers/2412.02592)** | **[💻 Code](https://github.com/opendatalab/OHR-Bench)** | **[🌐 Project Page (OpenDataLab)](https://opendatalab.com/OpenDataLab/OHR-Bench)** ![framework](https://github.com/opendatalab/OHR-Bench/blob/main/figs/framework.png) This repository contains the official code of **OHR-Bench**, a benchmark designed to evaluate the cascading impact of OCR on RAG. # News - 2025.6.30: Updating the results of MongkeyOCR, Nanonets-OCR-s and Azure Document Intelligence. - 2025.6.26: OHR-Bench has been accepted by ICCV 2025! # Overview - **PDF, gt structured data and Q&A datasets: [[🤗 Hugging Face](https://huggingface.co/datasets/opendatalab/OHR-Bench)] `pdfs.zip`, `data/retrieval_base/gt`, `data/qas_v2.json`**. It includes 8500+ unstructured PDF pages from 7 domains, including Textbook, Law, Finance, Newspaper, Manual, Academic and Administration and 8498 Q&A datasets sourced from 5 key components for OCR in document parsing, including plain text, table, formula, chart and reading order. Each PDF page is equipped with a human-verified ground truth structured data. - **Perturbed data with OCR errors: [[🤗 Hugging Face](https://huggingface.co/datasets/opendatalab/OHR-Bench)] `formatting_noise_[mild/moderate/severe]` and `semantic_noise_[GOT/MinerU/Qwen2.5-VL-72B]_[mild/moderate/severe]`**. In order to conduct in-depth analysis of the OCR's impact on RAG, OHR-Bench identifies *Semantic Noise* and *Formatting Noise* and introduce them with mild, moderate and severe perturbation based on real-world OCR errors. - **Evaluation framework: [[Github opendatalab/OHR-Bench](https://github.com/opendatalab/OHR-Bench)]**. We provide a RAG evaluation framework to assess the impact of OCR processed structured data and our perturbed data on RAG including retrieval, generation and overall performance. ## Evaluation Results <table> <thead> <tr> <th></th> <th>OCR</th> <th colspan="6">Retrieval</th> <th colspan="6">Generation</th> <th colspan="6">Overall</th> </tr> <tr> <th></th> <th>E.D.↓</th> <th>TXT↑</th> <th>TAB↑</th> <th>FOR↑</th> <th>CHA↑</th> <th>RO↑</th> <th>ALL↑</th> <th>TXT↑</th> <th>TAB↑</th> <th>FOR↑</th> <th>CHA↑</th> <th>RO↑</th> <th>ALL↑</th> <th>TXT↑</th> <th>TAB↑</th> <th>FOR↑</th> <th>CHA↑</th> <th>RO↑</th> <th>ALL↑</th> </tr> </thead> <tbody> <tr> <td>Ground Truth</td> <td>-</td> <td>81.6</td> <td>69.8</td> <td>75.2</td> <td>70.3</td> <td>9.8</td> <td>70.4</td> <td>49.3</td> <td>46.0</td> <td>34.0</td> <td>47.0</td> <td>28.2</td> <td>43.8</td> <td>44.9</td> <td>34.6</td> <td>28.0</td> <td>32.9</td> <td>18.7</td> <td>36.0</td> </tr> <tr> <td colspan="20">Pipeline-based OCR</td> </tr> <tr> <td>MinerU-0.9.3</td> <td>0.24</td> <td>68.1</td> <td>48.6</td> <td>51.3</td> <td>16.5</td> <td>5.9</td> <td>50.5</td> <td>45.7</td> <td>39.3</td> <td>28.6</td> <td>9.7</td> <td>29.5</td> <td>36.6</td> <td>41.4</td> <td>28.5</td> <td>23.0</td> <td>9.3</td> <td>17.8</td> <td>29.9</td> </tr> <tr> <td>Marker-1.2.3</td> <td>0.28</td> <td>75.5</td> <td>58.2</td> <td>55.5</td> <td>20.0</td> <td>5.9</td> <td>57.0</td> <td>44.4</td> <td>37.8</td> <td>27.8</td> <td>10.9</td> <td>26.2</td> <td>35.9</td> <td>40.1</td> <td>28.1</td> <td>22.3</td> <td>10.0</td> <td>16.2</td> <td>29.4</td> </tr> <tr> <td>Azure</td> <td>-</td> <td>78.0</td> <td>59.4</td> <td>55.2</td> <td>45.2</td> <td>5.8</td> <td>60.6</td> <td>41.0</td> <td>37.1</td> <td>27.1</td> <td>22.9</td> <td>27.3</td> <td>35.0</td> <td>37.5</td> <td>28.1</td> <td>22.6</td> <td>15.1</td> <td>17.5</td> <td>28.9</td> </tr> <tr> <td colspan="20">End-to-end OCR</td> </tr> <tr> <td>GOT</td> <td>0.27</td> <td>62.5</td> <td>41.1</td> <td>49.0</td> <td>17.4</td> <td>3.7</td> <td>45.8</td> <td>37.5</td> <td>28.5</td> <td>24.1</td> <td>8.5</td> <td>7.1</td> <td>27.8</td> <td>35.3</td> <td>22.9</td> <td>20.1</td> <td>8.2</td> <td>5.3</td> <td>24.5</td> </tr> <tr> <td>Nougat</td> <td>0.34</td> <td>59.5</td> <td>32.8</td> <td>44.3</td> <td>11.3</td> <td>4.4</td> <td>41.2</td> <td>36.6</td> <td>22.9</td> <td>22.9</td> <td>6.4</td> <td>6.9</td> <td>25.5</td> <td>33.5</td> <td>18.4</td> <td>19.4</td> <td>5.8</td> <td>3.6</td> <td>14.5</td> </tr> <tr> <td colspan="20">Vision-Language Model for OCR</td> </tr> <tr> <td>Qwen2.5-VL-72B</td> <td>0.18</td> <td>75.1</td> <td>60.0</td> <td>60.0</td> <td>38.2</td> <td>5.3</td> <td>59.6</td> <td>44.3</td> <td>42.1</td> <td>31.8</td> <td>27.0</td> <td>11.6</td> <td>37.5</td> <td>40.6</td> <td>31.1</td> <td>26.1</td> <td>19.0</td> <td>8.8</td> <td>31.1</td> </tr> <tr> <td>InternVL2.5-78B</td> <td>0.28</td> <td>68.6</td> <td>57.9</td> <td>55.6</td> <td>45.1</td> <td>2.7</td> <td>56.2</td> <td>41.7</td> <td>41.8</td> <td>29.0</td> <td>33.6</td> <td>3.3</td> <td>35.8</td> <td>38.2</td> <td>31.0</td> <td>23.3</td> <td>22.9</td> <td>3.1</td> <td>29.6</td> </tr> <tr> <td>olmOCR-7B-0225-preview</td> <td>0.21</td> <td>72.5</td> <td>58.4</td> <td>55.4</td> <td>24.8</td> <td>5.0</td> <td>56.6</td> <td>44.8</td> <td>40.5</td> <td>30.4</td> <td>19.0</td> <td>8.4</td> <td>36.0</td> <td>40.6</td> <td>30.3</td> <td>23.7</td> <td>12.8</td> <td>7.1</td> <td>29.6</td> </tr> <tr> <td>MonkeyOCR</td> <td>-</td> <td>74.6</td> <td>56.5</td> <td>55.5</td> <td>16.5</td> <td>5.7</td> <td>55.9</td> <td>40.3</td> <td>36.5</td> <td>25.9</td> <td>7.9</td> <td>25.0</td> <td>32.8</td> <td>35.4</td> <td>27.3</td> <td>20.7</td> <td>6.5</td> <td>16.3</td> <td>26.7</td> </tr> <tr> <td>Nanonets-OCR-s</td> <td>-</td> <td>71.8</td> <td>59.8</td> <td>57.4</td> <td>43.7</td> <td>4.4</td> <td>58.3</td> <td>38.2</td> <td>36.3</td> <td>28.0</td> <td>25.7</td> <td>7.8</td> <td>32.4</td> <td>34.9</td> <td>27.6</td> <td>22.7</td> <td>18.6</td> <td>7.1</td> <td>27.2</td> </tr> </tbody> </table> Notes: The subpar performance of Azure Document Intelligence may be related to the use of Llama3.1-8B as the generator. We are currently investigating this issue and plan to update the results using more advanced LLMs as generators. We evaluate the suitability of current OCR solutions for real-world RAG applications by conducting comprehensive experiments with our OHR-Bench. We report the generalized LCS or F1 of five types of evidence sources, including plain text (TXT), table (TAB), formula (FOR), chart (CHA), and reading order (RO). We derive conclusions as follows: - VLMs for OCR achieve the best overall performance. Employing Qwen2.5-VL-72B achieves the best performance across all OCR solutions. - All OCR solutions suffer performance degradation. Even the best solutions show a decrease of 14% F1-score in the overall evaluation, with greater losses in the retrieval and generation stages. # Getting Started ## Installation ```bash pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu121 ``` ## Dataset preparation ### OCR processed structured data To evaluate your RAG system on our benchmark, follow these steps: 1. **Download Perturbed Data**: Get the data with formatting and semantic noise from [the zip file in Hugging Face](https://huggingface.co/datasets/opendatalab/OHR-Bench/blob/main/retrieval.zip) and unzip it. Or use the load_dataset ("opendatalab/OHR-Bench") to get the relevant fields. 2. **Organize the Data**: Place the folders `retrieval_base/formatting_noise_[mild/moderate/severe]` and `retrieval_base/semantic_noise_[GOT/MinerU/Qwen2.5-VL-72B]_[mild/moderate/severe]` in the `data/retrieval_base` directory of this project. 3. **Run Evaluation**: Follow the instructions in [**Run Evaluation**](#run-evaluation). To evaluate your OCR results using this benchmark: 1. **Organize the Data**: Do OCR with your OCR models (PDFs available on [Hugging Face](https://huggingface.co/datasets/opendatalab/OHR-Bench)) and place the OCR processed structured data in the `data/retrieval_base` directory. Use the ground truth (`data/retrieval_base/gt`) data as an example. The sub-folder names indicate the domain of the parsed results, and each JSON file, named as the same of corresponding PDF files, should contain the corresponding parsed results. 2. **Run Evaluation**: Follow the instructions in [**Run Evaluation**](#run-evaluation). <details> <summary>Directory Structure</summary> ```bash retrieval_base/gt/ # We provide gt and MinerU processed structured data as illustration here ├── finance # Domain │ ├── 3M_2023Q2_10Q.json # Parsed results │ ├── ... ├── textbook ... ``` </details> <details> <summary>OCR Processed Data</summary> ```json [ { "page_idx": 0, // Page index "text": "...", // OCR processed structured data }, ... ] ``` </details> ### QA data The qa data is placed in `data/qas_v2.json`. Each JSON file should be structured as follows: <details> <summary>Q&A JSON</summary> ```json [ { "doc_name": "finance/JPMORGAN_2021Q1_10Q", // Document source "ID": "00073cc2-c801-467c-9039-fca63c78c6a9", // Unique ID "questions": "What was the total amount of nonaccrual loans retained as of March 31, 2021?", "answers": "842", "doc_type": "finance", // Q&A domain. "answer_form": "Numeric", // Answer format. "evidence_source": "table", // Evidence source. "evidence_context": "Nonaccrual loans retained $^{(\\mathrm{a})}$ & \\$ & 842 & \\$ & 689 & $22 \\%$", // Evidence. "evidence_page_no": 24 }, ... ] ``` </details> ## LLMs preparation In `src/configs`, configure your local LLM path or GPT API. ```python GPT_api_key = 'You KEY Here' # openai.api_key ... Qwen2_7B_local_path = 'Qwen/Qwen2-7B-Instruct' # download from Hugging Face or your local path ``` # Run Evaluation To evaluate your OCR results, follow the instructions in the Dataset Preparation section to organize your OCR data. ```bash # The first argument specifies which OCR results to use for evaluation. # The second argument specifies the retrievers or LLMs. # Args: Document source, LLM # Generation with gt bash shell/generation.sh gt qwen2_7b # Generation with mild semantic noise usi (OCR=MinerU) bash shell/generation.sh semantic_noise_MinerU_mild qwen2_7b # Args: Document source, retriver # Retrieval with gt bash shell/retrieval.sh gt bge-m3 # Retrieval with moderate semantic noise (OCR=MinerU) bash shell/retrieval.sh semantic_noise_MinerU_moderate bge-m3 # Args: Document source, retriver, LLM # End-to-end with gt bash shell/end2end.sh gt bge-m3 qwen2_7b # End-to-end with severe semantic noise (OCR=MinerU) bash shell/end2end.sh semantic_noise_MinerU_severe bge-m3 qwen2_7b ``` You can then use `exp_scripts/exp_show.ipynb` to view the results grouped by `domain` or `evidence_source`. # Acknowledgement The evaluation framework is based on [CRUD](https://github.com/IAAR-Shanghai/CRUD_RAG), thanks so much for this brilliant project. # Citation ``` @article{zhang2024ocr, title={OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation}, author={Junyuan Zhang and Qintong Zhang and Bin Wang and Linke Ouyang and Zichen Wen and Ying Li and Ka-Ho Chow and Conghui He and Wentao Zhang}, journal={arXiv preprint arXiv:2412.02592}, year={2024} } ``` # Copyright Statement The PDFs are collected from public online channels and community user contributions. Content that is not allowed for distribution has been removed. The dataset is for research purposes only and not for commercial use. If there are any copyright concerns, please contact OpenDataLab@pjlab.org.cn.

The OHR-Bench dataset is designed to evaluate the cascading impact of OCR on Retrieval-Augmented Generation (RAG). It includes over 8500 unstructured PDF pages from various domains and 8498 Q&A datasets, each PDF page is equipped with human-verified ground truth structured data. The dataset also introduces semantic noise and formatting noise at various levels to simulate real-world OCR errors.

提供机构：

opendatalab

搜集汇总

数据集介绍