JevOCRBench

# JevOCRBench [中文](README.zh-CN.md) · [Data terms](DATA_LICENSE.md) JevOCRBench is a benchmark for **OCR correctness decisions**. It tests whether a decision model can tell whether an OCR result is correct. JevOCRBench contains **2,267** Chinese and English records covering text, formulas, and tables. Each record mainly contains the source image, the OCR candidate text, a rendering of that candidate text, and a human-reviewed correctness label. Negatives come from real error outputs of frontier OCR models. ## Data information ### Distribution statistics  ### Data construction Images come from three public benchmarks of mainstream OCR models: [OmniDocBench v1.6](https://huggingface.co/datasets/opendatalab/OmniDocBench), [Real5-OmniDocBench](https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench), and [olmOCR-bench](https://huggingface.co/datasets/allenai/olmOCR-bench). Candidates were sampled from frontier OCR parsers such as MiniCPM-OCR 1.0 and PaddleOCR-VL-1.6, labeled by humans for correctness, then lightly filtered and balanced by category. ## Download and read Install and access the dataset with huggingface_hub: ```bash python -m pip install -U huggingface_hub ``` ```python import json from pathlib import Path from huggingface_hub import snapshot_download root = Path(snapshot_download( repo_id="openbmb/JevOCRBench", repo_type="dataset", )) with (root / "data/jevocrbench.jsonl").open(encoding="utf-8") as f: sample = json.loads(next(line for line in f if line.strip())) source_image = root / sample["image_path"] candidate_render = root / sample["render_image_path"] assert source_image.is_file() assert candidate_render.is_file() print(sample["item_id"], sample["ocr_text"]) ``` ## Data schema `data/jevocrbench.jsonl` contains one JSON object per line. Every row has exactly 14 fields in the order below. | Field | Type | Meaning | | --- | --- | --- | | `item_id` | integer | Sequential record ID from 0 to 2,266 and prediction join key. | | `label` | string | `pos` for a faithful candidate; `neg` for a candidate with a defect. | | `category` | string | `text`, `formula`, or `table`. | | `lang` | string | `en` or `zh`. | | `source` | string | One of 14 source-subset identifiers. | | `scenario` | string / null | Real5 acquisition condition; null outside Real5. | | `model` | string | Public display name of the OCR model/checkpoint that produced `ocr_text`. | | `page_id` | string | Source-local page identifier. | | `image_path` | string | Source-crop path relative to the repository root. | | `render_image_path` | string | Candidate-render path relative to the repository root. | | `ocr_text` | string | OCR candidate being verified. | | `error_type` | string / null | Primary error category for a classified negative. | | `error_types_secondary` | array / null | Zero to three additional error categories. | | `error_evidence` | string / null | Short Chinese description of the observed error. | Example: ```json { "item_id": 678, "label": "neg", "category": "text", "lang": "zh", "source": "omnidocbench_v1.6", "scenario": null, "model": "PaddleOCR-VL-1.6", "page_id": "newspaper_35bdaa11dec104ccbdc9e19ac8527d21_1.jpg", "image_path": "images/00654.png", "render_image_path": "renders/00679.png", "ocr_text": "## 莱集标准化", "error_type": "char_misrecognition", "error_types_secondary": [], "error_evidence": "聚焦二字被错识为莱集" } ``` All three error fields are null for positives. Definitions and boundary rules for all 11 error categories are in [`taxonomy/error_types.json`](taxonomy/error_types.json). ## Evaluation For every record, present these three inputs in order and evaluate them with our [rubic prompt](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/assets/minicpm-ocr/rubic_prompt.py): 1. source document crop; 2. raw OCR candidate text; 3. candidate rendering. A higher score from the decision model indicates a stronger tendency toward `pos`. All published results use these three inputs. Do not use labels, error types, sources, candidate models, or other metadata as inputs.  Evaluation code: ```python import json from sklearn.metrics import roc_auc_score recs = [json.loads(line) for line in open("data/jevocrbench.jsonl", encoding="utf-8")] # 2,267 records # Your verifier: one score per record (illustrative) # scores = [your_verifier(r["image_path"], r["ocr_text"], r["render_image_path"]) for r in recs] auroc = roc_auc_score([r["label"] == "pos" for r in recs], scores) ``` ## Repository layout ```text JevOCRBench/ ├── data/jevocrbench.jsonl ├── images/ # 2,188 source crops ├── renders/ # 2,267 per-record renders ├── viewer/ # Dataset Viewer Parquet with embedded images ├── taxonomy/error_types.json ├── assets/benchmark-results.png ├── assets/distribution-zh.png ├── assets/distribution-en.png ├── README.md ├── README.zh-CN.md ├── DATA_LICENSE.md ├── LICENSE # software only └── NOTICE ``` ## License and source terms The combined package uses `license: other` because different files and records have different terms: - JevOCRBench software: Apache License 2.0 under the repository-root [`LICENSE`](LICENSE); - eligible JevOCRBench-added metadata: Apache-2.0 only to the extent the contributors have authority to license it; - OmniDocBench-derived materials: upstream research-only, non-commercial terms; - Real5-derived materials: Real5 declares Apache-2.0, while inherited OmniDocBench annotations, source-page correspondences, and source-document terms remain applicable; - olmOCR-bench-derived database material: ODC-BY-1.0 with attribution. Underlying documents may carry additional rights. Read [DATA_LICENSE.md](DATA_LICENSE.md) and [NOTICE](NOTICE); those files and the upstream terms control over this summary.