Testing Scratch Programs Automatically

# Replication Package This is the replication package for our work on<br> "Testing Scratch Programs Automatically". The package contains our raw results and scripts for generating<br> the plots of the paper from the raw data. ## Abstract Block-based programming environments like Scratch foster engagement<br> with computer programming and are used by millions of young learners.<br> Scratch allows learners to quickly create entertaining programs and<br> games, while eliminating syntactical program errors that could<br> interfere with progress. However, functional programming errors may still lead to incorrect<br> programs, and learners and their teachers need to identify and<br> understand these errors. This is currently an entirely manual process. In this paper, we introduce a formal testing framework that describes<br> the problem of Scratch testing in detail. We instantiate this formal<br> framework with the Whisker tool, which provides automated and<br> property-based testing functionality for Scratch programs. Empirical evaluation on real student and teacher programs<br> demonstrates that Whisker can successfully test Scratch programs,<br> and automatically achieves an average of 95.25% code coverage. Although well-known testing problems such as test flakiness also<br> exist in the scenario of Scratch testing, we show that automated and<br> property-based testing can accurately reproduce and replace the<br> manually and laboriously produced grading efforts of a teacher, and<br> opens up new possibilities to support learners of programming in<br> their struggles. ## Contents The replication package is structured into two main directories: * 'data/':<br> raw data and scripts that have been used for collecting the data * 'scripts/':<br> scripts for generating the plots that are presented in the paper ### RAW data * 'data/teacher-data/'<br> data from the scratch workshop: sample solution and scores for student solutions * 'data/code-club-stats/'<br> block counts and input methods of the used Code Club projects * 'data/coverage`<br> code for measuring the coverage of automated input generation * 'data/coverage-results/'<br> coverage measurements on the Code Club projects * 'data/test/'<br> test suites for the projects of the Scratch workshop * 'data/test-results/'<br> test results from the test suites in 'data/test/' * 'data/time/'<br> Scratch programs for time measurement (10x the sample solution from 'data/teacher-data/`) * 'data/time-results/'<br> time measurements on the projects in 'data/time/' ## Reproducing the Plots ### Prerequisites We describe the process based on: * the R statistics package in version 3.5<br> * an Unix environment (Linux or MacOSX) Following R packages are required: * ggplot2<br> * dplyr<br> * viridis The package can be installed with the R command "install.packages". ### Generating the Plots Coverage (Figure 10) ./scripts/coverage.R The result is a set of "coverage-*.pdf" files Inconsistency (Figure 9) ./scripts/consistency.R The result is a set of "consistency-*.pdf" files Scatter Plots (Figure 8, Figure 11) ./scripts/scatter.R The result is a set of "scatter-*.pdf" files