Impact of Conversational Chatbots on Learning of University Students

To examine the impact of conversational chatbots on electronic learning in Bulgaria, an online survey was conducted using a Google Forms-based questionnaire from May 14 to May 31, 2023. The survey link was distributed via email and social media to student organizations and groups, resulting in 131 complete responses during the study period.
With the recent advancements in large language models, generative AI chatbots have gained popularity due to their enhanced capabilities in natural language processing. The majority of respondents (89%) reported having prior experience using conversational chatbots as an educational tool. The respondents evaluated their AI chatbot experience based on criteria such as frequency of use, perceived usefulness, trust and security, and facilitating conditions. The findings indicate that students who are aware of the capabilities and benefits of smart chatbots are more likely to use them frequently.
The second part of this study aims to evaluate the effectiveness of educational chatbots in handling university learning tasks, particularly in the field of mathematics. Two mathematics tasks were chosen and solved using different conversational chatbots. Here we can find information about cases where chatbots make mistakes and the types of errors. Further analysis provides valuable insights into the efficacy of chatbots in supporting mathematics learning at the university level. Among the seven evaluated conversational chatbots, ChatGPT Plus demonstrated the highest overall performance.