Unlocking the Potential of Socratic Tutoring Powered by Large Language Model

This pilot study introduced a conversational intelligent tutoring system named Socratic Playground for Learning (SPL) as a promising solution for simulating Socratic tutoring powered by GPT-4. It preliminarily investigated the effectiveness of Large Language Model-powered Socratic Tutoring (LLM-ST) in terms of learning gains and user experience. Results showed that: SPL could provide learners with high-quality content. Participants achieved an average proportional learning gain of 29.23%. Additionally, the acceptance of SPL was high, and the user experience was generally positive during learning: participants experienced higher levels of positive emotions and lower levels of negative emotions, with a notable reduction in negative emotions following the learning phase. Log analysis revealed that learners rarely expressed positive and negative emotions during interactions with SPL, and confusion was the primary emotion when expressed. Semi-structured interviews with participants indicated that SPL was effective in facilitating learning (e.g., supporting knowledge mastery and inspiring thinking), while also gathering valuable feedback for further improvements (e.g., simplifying learning phases and providing detailed guidance). These findings suggest that LLM-ST has the potential to facilitate knowledge acquisition and provide a positive user experience.