prometheus-eval/peerreview-bench

PeerReview Bench是一个用于评估AI在科学论文同行评审中应用的数据集，包含多个配置以支持不同的评估任务。数据集基于专家标注的评审项，涵盖科学论文的评审过程，所有数据仅用于评估而非训练。主要配置包括：1) reviewer：用于评估AI审稿人，每行代表一篇论文，包含论文ID、标题、内容及文件引用，通过重构论文文件并生成评审与专家标注进行对比。2) meta_reviewer：用于评估AI元审稿人，每行代表一个（论文、审稿人、评审项）组合，包含主次标注者的标签（如正确性、重要性、证据）以及一个10类别的标签，编码级联结果和每项指标的一致性。3) expert_annotation：用于统计分析和人类与AI评审相似性测量，每行包含一个标注者来源（主或次）的评审项，应用有效性规则过滤不完整数据。4) similarity_check：用于基准测试自动化相似性度量，包含164个评审项对，每个对具有二元标签（相似或不相似）和细粒度标签，用于评估评审项之间的相似性。5) submitted_papers：去重存储每篇论文的预印本文件，通过SHA256哈希引用。数据集支持多种使用场景，如AI审稿人生成评审、AI元审稿人标注、相似性度量等，并遵循CC-BY-4.0许可。数据规模包括数千个示例，例如expert_annotation配置有3881个示例。

PeerReview Bench is a dataset for evaluating AI applications in scientific paper peer review, containing multiple configurations for different evaluation tasks. The dataset is based on expert-annotated review items from scientific papers, organized for complementary evaluation purposes, with all data intended for evaluation, not training. Key configurations include: 1) reviewer: For evaluating AI reviewers, with one row per paper, including paper ID, title, content, and file references, used to reconstruct paper files and compare generated reviews to ground-truth expert annotations. 2) meta_reviewer: For evaluating AI meta-reviewers, with one row per (paper, reviewer, review_item) combination, including per-annotator labels (e.g., correctness, significance, evidence) and a collapsed label of 10 classes that encodes cascade outcomes and metric agreement. 3) expert_annotation: For statistical analysis and human-vs-AI review similarity measurement, with one row per (paper, reviewer, review_item, annotator_source), applying validity rules to filter incomplete data. 4) similarity_check: For benchmarking automated similarity metrics, containing 164 (paper, review item A, review item B) tuples with binary and fine-grained labels to assess similarity between review items. 5) submitted_papers: Deduplicated blob storage for preprint files, referenced via SHA256 content hashes. The dataset supports various use cases such as AI reviewer generation, AI meta-reviewer labeling, and similarity measurement, under the CC-BY-4.0 license. Data scales include thousands of examples, e.g., expert_annotation has 3881 examples.