Supporting data for "Over-Optimization of Academic Publishing Metrics: Observing Goodhart's Law in Action"

The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which "when a measure becomes a target, it ceases to be a good measure.'' In this study, we analyzed over 120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of over 2600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Academic publishing has changed considerably; now we need to reconsider how we measure success.

学术出版领域正经历深刻变革，每年刊发的学术成果数量持续攀升，出版模式亦不断迭代更新。然而，用以衡量学术成就的核心指标——如发文量、总引用数与影响因子（impact factor）——数十年来始终未有实质性调整。近期研究更指出，此类指标已逐渐异化为追逐目标，且契合古德哈特定律（Goodhart's Law）：当一项衡量指标沦为追逐目标时，它便不再能作为有效的衡量标准。本研究针对逾1.2亿篇学术论文展开分析，旨在探究过去一个世纪以来学术出版领域的演化历程，并针对生物学这一细分学科领域进行了深入剖析。研究结果表明，基于引用的各类衡量指标其有效性正持续受损，实用价值也日渐削弱。具体而言，作者署名列表日趋冗长、单篇论文篇幅缩减叠加发文量激增，使得发文量不再具备优质衡量指标的属性。诸如总引用数与h指数（h-index）这类基于引用的指标，同样受到论文数量泛滥、自引行为频发以及参考文献列表变长的负面影响。期刊影响因子等评价指标也已不再适用，这源于顶级期刊刊发的论文数量大幅攀升，尤其是来自同一作者群体的论文占比居高不下。此外，通过对超过2600个研究领域的属性进行分析，我们发现基于引用的指标并不适于比较不同研究领域，乃至同一院系内的研究者。学术出版格局已发生翻天覆地的变化，如今我们亟需重新审视学术成就的科学衡量方式。