Delphi Study: Exploring the Implications of Large Language Models on the Science System

Sample description: Our target audience consisted of researchers working in the fields of science, technology, and society with a specific interest in Large Language Models (LLMs).

Collection method: Participants were recruited through the professional and personal networks of the authors, as well as the Alexander von Humboldt Institute (HIIG), using a combination of generic emails via LimeSurvey and personal contacts.

Description. The aim of this study was to explore the impact of large language models, specifically ChatGPT, on scholarly practice and academic writing, targeting researchers and experts in the fields of artificial intelligence, science, and technology who publish their research and scientific work. The two-stage Delphi survey sought to identify and assess the potential opportunities and challenges associated with the use of ChatGPT in academic work and scientific writing, with a specific focus on research impact rather than university teaching. Phase 1 yielded 72 responses, while Phase 2 had 52 responses.

To conduct our analysis, we developed two distinct codebooks (see Files ChatGPT Delphi Codebook Phase 1.csv and ChatGPT Delphi Codebook Phase 2.csv) for the Delphi study. The first codebook was created by examining approximately half of the responses, extracting relevant information, and generating codes through inductive reasoning. We then categorized and developed subcodes based on these initial codes, assigning them to each participant's answers using deductive reasoning. For example, when addressing the potential applications of ChatGPT and other language models (LLMs), we identified six subcategories with precise definitions and illustrative examples. The analysis in Phase 1 led to the formulation of ranking questions for Phase 2, focusing on determining the most frequently utilized applications of ChatGPT and other LLMs based on the established codes.

During Phase 2, we introduced two additional open-ended questions to explore the impact of ChatGPT and LLMs on the scientific system and society, aiming to envision future scenarios. The analysis of these questions in the second codebook followed a similar approach to Phase 1, including inductive reasoning for code generation and deductive reasoning for assigning codes to the answers. We observed overlapping codes with the Phase 1 codebook and assigned them to the second codebook. Additionally, we noted a shift in the connotation of certain answers from neutral in Phase 1 to being perceived as either positive or negative consequences of ChatGPT and other LLMs. This observation prompted the bifurcation of specific codes to capture the nuanced perspectives. For instance, applications such as reducing administrative tasks initially seen as valuable aids for researchers were sometimes viewed as potential causes for job replacement, implying negative outcomes.

For detailed information on the analytical approach employed, including references to these methodologies, please refer to the methodology chapter in the official publication.

Content

Questionaire-ChatGPT-Delphi-Phase1-Limesurvey-Export.pdf – This file file is an exported version of the Phase 1 questionnaire from Limesurvey. It includes the description, socio demographic questions, content questions, and a request for participant naming.

Questionaire-ChatGPT-Delphi-Phase2-Limesurvey-Export.pdf – This file file is an exported version of the Phase 2 questionnaire from Limesurvey. It includes the description, socio demographic questions, content questions, and a request for participant naming.

ChatGPT Delphi - Results Phase 1.pdf – This file contains the responses and corresponding questions from Phase 1 of the Delphi study. The responses provided by the participants are in the form of open-ended answers. As part of this publication, we have ensured the anonymity of the participants.

ChatGPT Delphi - Results Phase 2. pdf – This file contains the responses and corresponding questions from Phase 2 of the Delphi study.  It encompasses the ranking answers provided by the participants, as well as two open-ended answers. To maintain anonymity consistently, all participants have been anonymized again in this publication of our results.

ChatGPT Delphi Codebook Phase 1.pdf – This file contains the Phase 1 codebook, which presents the primary codes, their respective subcodes, detailed definitions, and noteworthy examples.

ChatGPT Delphi Codebook Phase 2.pdf – This file contains the Phase 1 codebook, which provides a comprehensive overview of the primary codes within the given scenario. It includes their corresponding subcodes, detailed definitions, and notable examples to enhance understanding and interpretation.