"Towards Reliable Automatic Peer Review Generation and Evaluation"

"This dataset supports the research presented in the manuscript \"Towards Reliable Automatic Peer Review Generation and Evaluation via Analyzing Atomic Opinion between Peer-Review and Meta-Review\". The rapid advancement of AI research has led to an explosive growth in paper submissions, significantly challenging the traditional peer review process. Recently, Automatic Peer Review Generation (APRG) based on Large Language Models (LLMs) has gained increasing attention due to its potential to assist authors in iteratively improving paper quality and enhance reviewer efficiency, thereby alleviating peer review pressure. Existing works have demonstrated that fine-tuning LLMs on human-written peer reviews exhibits promising results. However, relying on subjective peer reviews while neglecting authoritative meta-reviews renders APRG training and evaluation unreliable, impeding further progress. To address this, we propose a novel approach leveraging structured Meta-Review Atomic opinions for high-quality data construction and evaluation. Specifically, we decompose peer reviews and meta-reviews into fine-grained Atomic Review Opinions (AROs). By aligning peer review AROs with meta-reviews to mitigate biases and conflicts, we construct a high-quality reference dataset. Furthermore, utilizing these curated opinions significantly enhances the reliability and interpretability of APRG evaluation. Building on this framework, we develop a comprehensive APRG suite comprising the ReviewTrain dataset, the ReviewEval benchmark, and the ReviewScore metric. Additionally, we train ReviewLLM utilizing the curated ReviewTrain corpus. Extensive experiments demonstrate that: (1) our ReviewScore significantly outperforms existing automatic metrics for peer review quality evaluation; (2) automatic evaluations on ReviewEval and DeepReview-Bench, along with human A\/B testing, verify that ReviewLLM significantly surpasses state-of-the-art baselines in both peer review quality and decision accuracy; and (3) comprehensive analyses further validate the high quality of our dataset, as well as the model\u2019s scalability and robustness against adversarial attacks. The resources of our developed datasets and codes have been publicly released1 to support future research in APRG2."