RPC-Bench

<div align="center"> # RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension </div> <p align="center"> 🌐 <a href="https://rpc-bench.github.io/" target="_blank">Project Page</a> • 💻 <a href="https://github.com/zai-org/RPC-Bench" target="_blank">GitHub</a> • 📖 <a href="https://arxiv.org/abs/2601.14289" target="_blank">Paper</a> </p> <div align="center"> <img src="assets/pipeline.png" width="100%" /> </div> RPC-Bench is a fine-grained benchmark for research paper comprehension. It is built from review-rebuttal exchanges of high-quality academic papers and supports both text-only and visual evaluation through complementary paper representations. ## Data Structure RPC-Bench is organized into `train`, `dev`, and `test` subsets. Split assignments are recorded in `manifest.jsonl`, and the original split JSON files are provided in `split_metadata/` (`train.json`, `dev.json`, `test.json`). `md/` contains Markdown files parsed from each paper by MinerU. These files provide the text input for LLM-oriented evaluation. `parse/` contains the full MinerU parsing outputs for each paper, including structured layout and content artifacts. `pdf/` contains the original paper PDFs. `vlm/` contains page images rendered from the PDFs with PyMuPDF at 200 DPI for VLM-oriented evaluation. ```text RPC-Bench/ ├── README.md ├── manifest.jsonl ├── split_metadata/ │ ├── train.json │ ├── dev.json │ └── test.json ├── parse/ │ ├── train/ │ │ └── <paper_id>/ │ ├── dev/ │ │ └── <paper_id>/ │ └── test/ │ └── <paper_id>/ ├── md/ │ ├── train/ │ │ └── <paper_id>/ │ │ └── <paper_id>.md │ ├── dev/ │ │ └── <paper_id>/ │ │ └── <paper_id>.md │ └── test/ │ └── <paper_id>/ │ └── <paper_id>.md ├── pdf/ │ ├── train/ │ │ └── <paper_id>.pdf │ ├── dev/ │ │ └── <paper_id>.pdf │ └── test/ │ └── <paper_id>.pdf └── vlm/ ├── train/ │ └── <paper_id>/ ├── dev/ │ └── <paper_id>/ └── test/ └── <paper_id>/ ``` ## Practical Uses RPC-Bench can be used to try paper-centric systems that require broader document understanding rather than local snippet matching. - Research paper comprehension: try models on full-paper understanding, including core concepts, methods, and experimental findings. - Long-context evaluation: try whether longer context windows or long-context architectures improve document-level reasoning. - Multimodal reasoning: try models that combine textual evidence with page-level figures, tables, and diagrams in the original PDF layout. - RAG system diagnosis: try retrieval, chunking, and evidence-fusion strategies for paper-centric workflows beyond snippet-level retrieval accuracy. ## Citation ```bibtex @article{chen2026rpc, title={RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension}, author={Chen, Yelin and Zhang, Fanjin and Sun, Suping and Pang, Yunhe and Wang, Yuanchun and Song, Jian and Li, Xiaoyan and Hou, Lei and Zhao, Shu and Tang, Jie and others}, journal={arXiv preprint arXiv:2601.14289}, year={2026} } ```