Can ChatGPT be a good follower of academic paradigms? Research quality evaluations in conflicting areas of sociology

Data from the paper: Can ChatGPT be a good follower of academic paradigms? Research quality evaluations in conflicting areas of sociologyMike Thelwall, School of Information, Journalism and Communication, University of Sheffield, UK. Ralph Schroeder, Oxford Internet Institute, University of Oxford UK. Meena Dhanda, Visiting Professor, Department of Media and Communications, London School of Economics and Political Science, U.K. Purpose: It has become increasingly likely that Large Language Models (LLMs) will be used to score the quality of academic publications to support research assessment goals in the future. This may cause problems for fields with competing paradigms since there is a risk that one may be favoured, causing long term harm to the reputation of the other.Design/methodology/approach: To test whether this is plausible, study 1 uses 17 ChatGPTs to evaluate up to 100 journal articles from each of eight pairs of competing sociology paradigms (1490 altogether). Each article was assessed by prompting ChatGPT to take one of five roles: paradigm follower, opponent, antagonistic follower, antagonistic opponent, or neutral. Study 2 involved five pairs of more tightly defined paradigms.Findings: Articles were scored highest by ChatGPT when it followed the aligning paradigm, and lowest when it was told to devalue it and to follow the opposing paradigm. Broadly similar patterns occurred for most of the paradigm pairs. Follower ChatGPTs displayed only a small amount of favouritism compared to neutral ChatGPTs, but articles evaluated by an opposing paradigm ChatGPT had a substantial disadvantage in some cases.Research limitations: The data covers a single field and LLM.Practical implications: The results confirm that LLM instructions for research evaluation should be carefully designed to ensure that they are paradigm-neutral to avoid accidentally resolving conflicts between paradigms on a technicality by devaluing one side’s contributions.Originality/value: This is the first demonstration that LLMs can be prompted to show a partiality for academic paradigms.

本研究数据来源于论文：《ChatGPT能否成为学术范式的忠实追随者？社会学冲突领域的研究质量评估》。作者：迈克·塞尔沃尔（Mike Thelwall），英国谢菲尔德大学信息、新闻与传播学院；拉尔夫·施罗德（Ralph Schroeder），英国牛津大学牛津互联网研究所；米娜·丹达（Meena Dhanda），英国伦敦政治经济学院媒体与传播学系访问教授。

研究目的：大型语言模型（Large Language Models，LLMs）未来用于评分学术出版物质量以支撑科研评估目标的可能性与日俱增。这对于存在竞争范式的领域可能引发问题，因为存在某一范式被偏袒的风险，进而长期损害另一范式的学术声誉。

研究设计与方法：为验证该假设的合理性，研究1使用17个ChatGPT实例，对8组竞争社会学范式的各至多100篇期刊论文（总计1490篇）开展评估。每篇论文均通过提示ChatGPT扮演以下五种角色之一完成评估：范式追随者、反对者、对立式追随者、对立式反对者及中立者。研究2则涉及5组定义更严谨的范式。

研究结果：当ChatGPT扮演与论文所属范式匹配的追随者角色时，其给出的评分最高；当被要求贬低该范式并遵循对立范式时，评分最低。多数范式组均呈现大致相似的评分规律。相较于中立ChatGPT，范式追随者ChatGPT仅表现出微弱的偏袒倾向，但部分场景下，由对立范式ChatGPT评估的论文会面临显著劣势。

研究局限：本研究数据仅覆盖单一学科与一款大语言模型。

实践启示：研究结果证实，用于科研评估的大语言模型指令需经精心设计，确保其具备范式中立性，以避免因技术细节问题，意外消解范式间的学术冲突并贬低某一方的研究贡献。

原创性与价值：本研究首次证实，可通过提示大语言模型使其表现出对学术范式的偏向性。