The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

We conducted an experiment during the review process of the 2023 International Conference on Machine Learning (ICML), asking authors with multiple submissions to rank their papers based on perceived quality. In total, we received 1342 rankings, each from a different author, covering 2592 submissions. In this article, we present an empirical analysis of how author-provided rankings could be leveraged to improve peer review processes at machine learning conferences. We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings. Our analysis shows that these ranking-calibrated scores outperform the raw review scores in estimating the ground truth “expected review scores” in terms of both squared and absolute error metrics. Furthermore, we propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review, including supporting senior area chairs in overseeing area chairs’ recommendations, assisting in the selection of paper awards, and guiding the recruitment of emergency reviewers. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.本研究于2023年国际机器学习大会（International Conference on Machine Learning, ICML）的审稿流程期间开展了一项实验，邀请提交多篇投稿的作者根据主观感知的论文质量对其稿件进行排序。本次实验共回收1342份有效排序结果，均来自不同作者，覆盖2592篇投稿论文。本文针对作者提供的排序结果如何用于优化机器学习学术会议的同行审稿流程展开实证分析。本研究聚焦于保序校准机制（Isotonic Mechanism），该方法可借助作者提供的排序结果对原始审稿得分进行校准。分析结果表明，经排序校准后的得分在估计真实"预期审稿得分"时，无论是平方误差还是绝对误差指标上均优于原始审稿得分。此外，本文还提出了保序校准机制与作者提供的排序结果在同行审稿中的若干审慎低风险应用场景，包括协助高级领域主席监督领域主席的投稿推荐结果、辅助优秀论文奖项评选，以及指导应急审稿人招募工作。本文的补充材料已在线发布，其中包含可用于复现本研究成果的标准化材料说明。