Evaluating Knowledge Tracing Models for Competency Assessment in Software Engineering Education

Knowledge Tracing (KT) tracks students' learning progress to adapt learning content and feedback accordingly. Hence, it is a crucial part of educational technologies, particularly Intelligent Tutoring Systems (ITSs). Which KT model is most suitable depends on contextual factors, particularly the learning domain. While substantial prior work has covered KT in computer science education, few studies have provided open datasets and comprehensively compared KT models in the context of Software Engineering (SE) education. We provide an open dataset from a first-semester programming and SE course with about 600 students and compare a requirement-driven set of established KT models to identify suitable candidate models for an ITS for SE education. Based on ITS-oriented requirements (e.g., scalability, interpretability, support for repeated attempts, and items involving multiple competencies), we compare established KT models and Elo-based variants in terms of (i) predicting the next response within a semester and (ii) transfer between semesters as well as (iii) predicting results of final exam tasks based on interactions during the course. Our results show that an Elo-based model (M-Elo) achieves strong predictive performance while meeting most of our predefined requirements. Furthermore, our results indicate limited predictive power for final exam performance. The results provide valuable insights into KT in SE education and highlight models that potentially perform well in ITSs for SE education.