A Block-Based Testing Framework for Scratch

Block-based programming environments like SCRATCH are widely used in introductory programming courses. They facilitate learning pivotal programming concepts by eliminating syntactical errors, but logical errors that break the desired program behaviour are nevertheless possible. Finding such errors requires <i>testing</i>, i.e., running the program and checking its behaviour. In many programming environments this step can be automated by providing executable tests as code; in SCRATCH testing can only be done manually by clicking the green flag icon and observing the rendered stage. While this is arguably sufficient for learners, the lack of automated testing may be inhibitive for teachers having to provide feedback on their students’ solutions. In order to address this issue, we introduce a new category of blocks in SCRATCH that enables the creation of automated tests. This allows students and teachers alike to create tests for SCRATCH programs within the SCRATCH environment using familiar block-based programming logic. Furthermore, we extended the SCRATCH user interface with an accompanying test interface that facilitates the creation of test suites as well as batch processing sets of student solutions. We evaluated the SCRATCH test interface with 28 teachers, who created tests for a popular SCRATCH game and then used these tests to assess and provide feedback to student implementations of the same game. An overall accuracy of 0.93 of the teachers’ tests on all aspects of functionality of 21 student solutions demonstrates that teachers are able to create and use effective tests. Furthermore, a subsequent survey confirms that teachers consider the block-based test approach useful.