+5 Million Python & Bash Programming Submissions for 5 Courses & Grades for Computer-Based Exams over 3 academic years.

In Dublin City University, students learn how to code by taking a variety of programming modules. Students develop code algorithms for problems proposed by Faculty. Many of these courses or modules are delivered through a custom Virtual Learning Environment (VLE) built for the purpose of teaching and learning computer programming. This custom VLE enables students to access course information, material and slides for each module. In addition, our system integrates an automatic grading platform where students can verify their code submissions for programming exercises. Students typically develop solutions locally for laboratory sheets for the computer programming courses. Then, they submit their programs online to the automatic grading platform which runs a number of testcases specified by the lecturer on each exercise. This provides instant feedback to students based on the suite of testcases run and ultimately tells the student whether the program is considered correct or incorrect if any of the testcases fail. This information is invaluable to their learning and such a platform is needed to verify their programs work as expected.
The computer programming grading system has been used for several years on a variety of programming courses at our University. This allowed researchers and Faculty to gather a fine-grained digital footprint of students learning programming at our University. Recently, research in Learning Analytics has focused on Predictive Modelling and identifying those students having difficulties with course material, also in programming courses, and offering remediation, personalized feedback and interventions to students using Machine Learning techniques. Prior work has reported that customized notifications sent to students regarding their performance and offering resources such as further learning material, code solutions from peers in their class and university support services helped students to increase their differential performance and engagement on these programming courses. However, there is a limit to this prior work where most of the models use little or no programming work as features for the learning algorithms or feedback sent to students. In this work we explore different mechanisms to represent students’ code to predict its correctness and to better analyze students’ progress using their interactions which can be exploited to provide effective feedback and support better recommendations.
Every time a student submits a code solution for verification, the system stores the code submission, the student identifier, the IP used on the network for the upload, the results of the testcases run with inputs and outputs, the course the submission belongs to, the exercise and the task name the student is attempting by using the submission’s filename. In total, we collected more than half a million programming submissions (591,707) for 666 students from 5 Python programming courses over 3 academic years.