Data_Sheet_1_Embracing LLM Feedback: the role of feedback providers and provider information for feedback effectiveness.docx

Feedback is an integral part of learning in higher education and is increasingly being provided to students via modern technologies like Large Language Models (LLMs). But students’ perception of feedback from LLMs vs. feedback from educators remains unclear even though it is an important facet of feedback effectiveness. Further, feedback effectiveness can be negatively influenced by various factors; For example, (not) knowing certain characteristics about the feedback provider may bias a student’s reaction to the feedback process. To assess perceptions of LLM feedback and mitigate the negative effects of possible biases, this study investigated the potential of providing provider-information about feedback providers. In a 2×2 between-subjects design with the factors feedback provider (LLM vs. educator) and provider-information (yes vs. no), 169 German students evaluated feedback message and provider perceptions. Path analyses showed that the LLM was perceived as more trustworthy than an educator and that the provision of provider-information led to improved perceptions of the feedback. Furthermore, the effect of the provider and the feedback on perceived trustworthiness and fairness changed when provider-information was provided. Overall, our study highlights the importance of further research on feedback processes that include LLMs due to their influential nature and suggests practical recommendations for designing digital feedback processes.

反馈是高等教育学习过程中的一个重要组成部分，且随着现代技术的进步，如大语言模型（LLM）等技术的应用，反馈信息正日益通过这些技术手段传达给学生。然而，学生对来自LLM的反馈与来自教育者的反馈之间的感知仍不明确，尽管这关乎反馈效果的显著方面。此外，反馈效果可能受到多种因素的负面影响；例如，对反馈提供者某些特征的（不）了解可能会使学生对反馈过程的反应产生偏差。为了评估学生对LLM反馈的感知并减轻可能的偏见带来的负面影响，本研究探讨了提供反馈提供者信息（是/否）的潜在效果。在2×2的被试间设计中，以反馈提供者（LLM与教育者）和提供者信息（是/否）为因素，169名德国学生对反馈信息和提供者感知进行了评估。路径分析表明，LLM被认为比教育者更具可信度，而提供提供者信息则导致了反馈感知的改善。此外，当提供提供者信息时，提供者和反馈对感知可信度和公平性的影响发生了变化。总体而言，我们的研究突出了进一步研究包含LLM的反馈过程的重要性，因其影响力显著，并提出了设计数字化反馈过程的实践建议。