XL-DocBench

<div align="center"> <h1>XL-DocBench</h1> <p> Evidence-grounded reasoning across hundreds or thousands of pages.<br> Fully verified by 194 human experts. </p> <p> Hongchen Wei^1,†,‡, Yuanzhe Wang^2,†,‡, Bei Liu^2,*, Yifan Yang², Qi Dai², Ruichun Ma², Kai Qiu², Yunsheng Li²,<br> Dongdong Chen², Chong Luo², Zhenzhong Chen¹, Baining Guo² </p> <p> ¹Wuhan University &nbsp; ²Microsoft &nbsp; ^†Equal contribution &nbsp; ^‡Work done during an internship at MSRA &nbsp; ^*Project leader </p> <p> <a href="https://officeintelligence.github.io/xl-docbench/"><b>Project Page</b></a> · <a href="https://arxiv.org/abs/2608.00036"><b>Paper</b></a> · <a href="https://officeintelligence.github.io/xl-docbench/#leaderboard"><b>Live Leaderboard</b></a> </p> <p> <a href="https://arxiv.org/abs/2608.00036"><img src="https://img.shields.io/badge/arXiv-2608.00036-b31b1b.svg" alt="arXiv"></a> <img src="https://img.shields.io/badge/questions-1,345-3b5b92.svg" alt="1,345 questions"> <img src="https://img.shields.io/badge/documents-292-6a4c93.svg" alt="292 documents"> <img src="https://img.shields.io/badge/human_verified-194_experts-2d6a4f.svg" alt="Verified by 194 experts"> </p> </div> ## TL;DR This is the **conservative XL-DocBench release**. It contains **1,345 QA rows over 292 documents** after removing every question that touches an exact source URL marked `RAG: Not Approved`. ## This release **292 documents** · **1,191 single-document QA** · **154 cross-document QA** · **1,345 total QA** - `data/documents.jsonl`: retained document metadata and source URLs. - `data/qa_single_doc.jsonl`: retained single-document questions. - `data/qa_cross_doc.jsonl`: retained cross-document questions. - `manifest.json`: recomputed release statistics. - `results/scores.jsonl`: per-question scores for the 13 reproducible systems. - `results/summary.json`: aggregate scores for the same systems. - `code/quickstart.py`: one-command data and evaluator smoke test. - `code/evaluate.py`: self-contained deterministic evaluator. The filter uses exact URL matching. This variant addresses exact `Not Approved` RAG rows only and does not interpret separate `Portions Approved` entries. ## About XL-DocBench XL-DocBench asks systems to **find the evidence, combine all required support, apply the right rule, and know when to abstain**. This strict release contains 1,345 expert-verified questions from six professional domains. Among 1,280 records with parseable historical human page annotations, 975 (76.2%) use multiple evidence pages. Released supporting evidence is multimodal for 429 questions (31.9%), 154 questions (11.4%) use cross-document contexts, and 188 require a `None` answer. Full-series contexts reach 2,935 pages. ## Reproducible strict results All systems below have complete scores for the 1,345 retained IDs and use the hardened evaluator shipped in this release. | Rank | System | Input | Accuracy ↑ | Token F1 ↑ | ANLS ↑ | |---:|---|:---:|---:|---:|---:| | 1 | **GPT-5.4** | OCR | **38.36** | **39.70** | **34.17** | | 2 | SimpleDoc + GPT-5.4 | Agent | 36.21 | 33.62 | 24.30 | | 3 | Kimi-K2.5 | OCR | 36.06 | 38.24 | 32.64 | | 4 | MDocAgent + GPT-5.4 | Agent | 31.82 | 30.58 | 22.64 | | 5 | GPT-5.2 | OCR | 31.15 | 33.69 | 28.51 | | 6 | DeepSeek-V3.2 | OCR | 29.67 | 32.99 | 28.91 | | 7 | Qwen3.5-4B | OCR | 29.00 | 32.23 | 28.13 | | 8 | DeepRead + GPT-5.4 | Agent | 27.73 | 26.99 | 21.17 | | 9 | GPT-5.4 | Img | 26.77 | 29.76 | 26.33 | | 10 | Kimi-K2.5 | Img | 26.02 | 27.59 | 23.04 | | 11 | GPT-5.2 | Img | 21.71 | 24.95 | 21.92 | | 12 | Qwen3.5-4B | Img | 20.52 | 20.86 | 18.26 | | 13 | Qwen3.5-9B | Img | 20.22 | 22.69 | 20.18 | ## Per-question scores Source documents are referenced by public URLs rather than redistributed. Because some URLs may change or become unavailable over time, we also provide the benchmark scores for every retained question ID. This gives future users a stable comparison point even when a source URL is temporarily unavailable. - `results/scores.jsonl`: one row for each of the 1,345 question IDs. - `results/summary.json`: aggregate Accuracy, Token F1, and ANLS. Scores are stored on a 0-to-1 scale. ## Quick start Validate the complete release and run a five-question evaluation fixture: ```bash uv run --no-project python code/quickstart.py ``` Load all three JSONL tables with the included standard-library example: ```bash uv run --no-project python code/examples/load_data.py ``` ## Evaluation Try the bundled five-question example: ```bash uv run --no-project python code/evaluate.py \ --gold-files code/examples/gold_sample.jsonl \ --predictions code/examples/predictions_sample.jsonl ``` For a complete run, provide one prediction per question: ```jsonl {"question_id": "adubench_single_000001", "prediction": "the biggest single risk to human health worldwide"} {"question_id": "adubench_cross_000001", "prediction": "macroprudential measures"} ``` ```bash uv run --no-project python code/evaluate.py \ --predictions predictions.jsonl \ --output eval_report.json \ --per-question-csv per_question.csv ``` The evaluator reports rule-based **Accuracy**, token-level **F1**, and **ANLS**. Missing, failed, and unparsable predictions count as incorrect unless `--ignore-missing` is enabled. ## Record structure ```text question ├── answer: value + format + verification rule ├── document / documents │ ├── document_id + public URL │ ├── evidence_pages: one-based PDF/release page indices │ └── evidence_items: annotator locator + page references + excerpt kind └── metadata: domain + difficulty + reasoning type + answerability ``` `evidence_pages` locate pages in the released PDF context. Evidence-item `pages` preserve annotator-supplied page references; printed pagination can differ from PDF indices. The `page_numbering` and `evidence_kind` fields make those cases explicit. Row-level `unassigned_evidence_items` are retained as provenance and are not counted as released supporting evidence. PDF binaries and local filenames are not included. Source documents remain subject to their original licenses and terms; this release does not grant redistribution rights for third-party PDFs. ## Citation ```bibtex @article{wei2026xldocbench, title = {XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding}, author = {Wei, Hongchen and Wang, Yuanzhe and Liu, Bei and Yang, Yifan and Dai, Qi and Ma, Ruichun and Qiu, Kai and Li, Yunsheng and Chen, Dongdong and Luo, Chong and Chen, Zhenzhong and Guo, Baining}, journal = {arXiv preprint arXiv:2608.00036}, year = {2026} } ```