TrustAIRLab/PeerCheck

--- license: apache-2.0 task_categories: - text-generation - text-classification language: - en tags: - peer-review - llm - evaluation - text-generation - ai-generated-text - review-quality pretty_name: >- PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality size_categories: - 1K<n<10K --- # **PeerCheck**: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality ## Dataset Summary **PeerCheck** is a framework for studying and improving the quality of LLM-generated academic peer reviews. It contains both **human-written reviews** and **LLM-generated reviews** for the same research papers, enabling direct comparison between human and LLM-generated reviewers. The dataset is used to support research on: - LLM-generated peer review; - Review quality evaluation; - Human–LLM alignment in scientific assessment. --- ## Dataset Description ### Data Sources The dataset is constructed from publicly available peer reviews collected from: - ICLR 2024 / 2025 - NeurIPS 2024 (via OpenReview) For each paper: - Human-written reviews are collected; - LLM-based reviews, including GPT-4o, Claude-3.7-Sonnet, and DeepSeek-V3, generate corresponding reviews. All data are processed to remove personal information and ensure quality. --- ## Limitations - Focused on machine learning papers; - LLM-generated reviews may contain biases or inaccuracies; - Not intended to replace human peer review. --- ## Ethical Considerations All data are collected from publicly available sources (OpenReview). Personal identifiers are removed or anonymized. This study was reviewed and approved by the Ethics Review Board (No. 25-12-6) of our institution. This dataset is intended **for research purposes only** and should not be used to automate real-world peer review decisions. --- ## Citation If you use this dataset, please cite: ```bibtex @article{peercheck2025, author={Zeyuan Chen and Ziqing Yang and Yihan Ma and Michael Backes and Yang Zhang}, title= {{PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality}}, booktitle = {{Findings of the Association for Computational Linguistics: ACL (ACL Findings)}}, publisher = {ACL}, year = {2026} }