Unlocking the Potential of Socratic Tutoring Powered by Large Language Model

This pilot study introduced a conversational intelligent tutoring system named Socratic Playground for Learning (SPL) as a promising solution for simulating Socratic tutoring powered by GPT-4. It preliminarily investigated the effectiveness of Large Language Model-powered Socratic Tutoring (LLM-ST) in terms of learning gains and user experience. Results showed that: SPL could provide learners with high-quality content. Participants achieved an average proportional learning gain of 29.23%. Additionally, the acceptance of SPL was high, and the user experience was generally positive during learning: participants experienced higher levels of positive emotions and lower levels of negative emotions, with a notable reduction in negative emotions following the learning phase. Log analysis revealed that learners rarely expressed positive and negative emotions during interactions with SPL, and confusion was the primary emotion when expressed. Semi-structured interviews with participants indicated that SPL was effective in facilitating learning (e.g., supporting knowledge mastery and inspiring thinking), while also gathering valuable feedback for further improvements (e.g., simplifying learning phases and providing detailed guidance). These findings suggest that LLM-ST has the potential to facilitate knowledge acquisition and provide a positive user experience.本先导性研究介绍了一款名为“学习型苏格拉底互动平台（Socratic Playground for Learning, SPL）”的会话型智能辅导系统，将其作为依托GPT-4模拟苏格拉底式辅导的极具潜力的解决方案。本研究初步探究了基于大语言模型（Large Language Model, LLM）的苏格拉底式辅导（Large Language Model-powered Socratic Tutoring, LLM-ST）在学习收获与用户体验维度的有效性。研究结果显示：SPL可为学习者提供高质量的学习内容。参与研究的学习者平均学习增益比例达29.23%。此外，学习者对SPL的接受度较高，整体学习体验良好：学习者在学习过程中体验到更高水平的积极情绪与更低水平的消极情绪，且学习阶段结束后消极情绪出现了显著降低。日志分析结果表明，学习者在与SPL交互过程中极少表达正负向情绪，当表达情绪时，困惑是最主要的情绪类型。对参与者的半结构化访谈结果表明，SPL可有效促进学习（例如辅助学习者掌握知识、激发思考），同时也为后续优化收集到了宝贵的改进建议，例如简化学习流程、提供更细致的指导。上述研究结果表明，LLM-ST具备促进知识获取并带来良好用户体验的潜力。