Testing Scratch Programs Automatically

# Replication Package This is the replication package for our work on<br> "Testing Scratch Programs Automatically". The package contains our raw results and scripts for generating<br> the plots of the paper from the raw data. ## Abstract Block-based programming environments like Scratch foster engagement<br> with computer programming and are used by millions of young learners.<br> Scratch allows learners to quickly create entertaining programs and<br> games, while eliminating syntactical program errors that could<br> interfere with progress. However, functional programming errors may still lead to incorrect<br> programs, and learners and their teachers need to identify and<br> understand these errors. This is currently an entirely manual process. In this paper, we introduce a formal testing framework that describes<br> the problem of Scratch testing in detail. We instantiate this formal<br> framework with the Whisker tool, which provides automated and<br> property-based testing functionality for Scratch programs. Empirical evaluation on real student and teacher programs<br> demonstrates that Whisker can successfully test Scratch programs,<br> and automatically achieves an average of 95.25% code coverage. Although well-known testing problems such as test flakiness also<br> exist in the scenario of Scratch testing, we show that automated and<br> property-based testing can accurately reproduce and replace the<br> manually and laboriously produced grading efforts of a teacher, and<br> opens up new possibilities to support learners of programming in<br> their struggles. ## Contents The replication package is structured into two main directories: * 'data/':<br> raw data and scripts that have been used for collecting the data * 'scripts/':<br> scripts for generating the plots that are presented in the paper ### RAW data * 'data/teacher-data/'<br> data from the scratch workshop: sample solution and scores for student solutions * 'data/code-club-stats/'<br> block counts and input methods of the used Code Club projects * 'data/coverage`<br> code for measuring the coverage of automated input generation * 'data/coverage-results/'<br> coverage measurements on the Code Club projects * 'data/test/'<br> test suites for the projects of the Scratch workshop * 'data/test-results/'<br> test results from the test suites in 'data/test/' * 'data/time/'<br> Scratch programs for time measurement (10x the sample solution from 'data/teacher-data/`) * 'data/time-results/'<br> time measurements on the projects in 'data/time/' ## Reproducing the Plots ### Prerequisites We describe the process based on: * the R statistics package in version 3.5<br> * an Unix environment (Linux or MacOSX) Following R packages are required: * ggplot2<br> * dplyr<br> * viridis The package can be installed with the R command "install.packages". ### Generating the Plots Coverage (Figure 10) ./scripts/coverage.R The result is a set of "coverage-*.pdf" files Inconsistency (Figure 9) ./scripts/consistency.R The result is a set of "consistency-*.pdf" files Scatter Plots (Figure 8, Figure 11) ./scripts/scatter.R The result is a set of "scatter-*.pdf" files# 复现包 本复现包对应我们题为"自动化测试Scratch程序（Testing Scratch Programs Automatically）"的研究工作。 本包包含我们的原始实验结果，以及用于基于原始数据生成论文配图的脚本。 ## 摘要 诸如Scratch的基于块的编程环境能够提升学习者对计算机编程的参与度，目前已有数百万青少年学习者使用该类环境。Scratch允许学习者快速创作富有趣味性的程序与游戏，同时规避了可能阻碍学习进度的语法类程序错误。 但功能性编程错误仍可能导致程序运行异常，学习者与教师需要手动识别并排查此类错误，目前这一过程完全依赖人工完成。 在本研究中，我们提出了一个形式化测试框架，对Scratch程序测试问题进行了详细的形式化描述。我们使用Whisker工具对该形式化框架进行了实例化，该工具可为Scratch程序提供自动化测试与基于属性的测试（property-based testing）功能。 针对真实学生与教师编写的程序开展的实证评估表明，Whisker能够有效完成Scratch程序的自动化测试，且平均代码覆盖率可达95.25%。 尽管Scratch程序测试场景中同样存在测试抖动（test flakiness）等广为人知的测试难题，但我们的研究表明，自动化测试与基于属性的测试能够精准复现并替代教师耗时费力完成的手工评分工作，同时为编程学习者的学习困境提供了新的可能性。 ## 包内容 本复现包包含两个主要目录： * `data/`：用于采集原始数据的原始数据集与相关脚本 * `scripts/`：用于生成论文中配图的脚本 ### 原始数据集 * `data/teacher-data/`：来自Scratch工作坊的数据集，包含示例程序解法与学生作品的评分结果 * `data/code-club-stats/`：所用Code Club项目的代码块统计数与输入方式信息 * `data/coverage/`：用于衡量自动化输入生成覆盖率的代码（原文此处疑似漏写斜杠，已按学术惯例补全） * `data/coverage-results/`：针对Code Club项目的覆盖率测试结果 * `data/test/`：Scratch工作坊项目的测试套件 * `data/test-results/`：`data/test/`中测试套件对应的测试结果 * `data/time/`：用于性能耗时测试的Scratch程序（规模为`data/teacher-data/`中示例解法的10倍） * `data/time-results/`：针对`data/time/`中项目的耗时测试结果 ## 生成论文配图 ### 前置依赖 本流程基于以下环境搭建： * 版本为3.5的R统计分析包 * Unix类操作系统（Linux或MacOSX） 所需依赖的R包如下： * ggplot2 * dplyr * viridis 可通过R命令`install.packages()`完成安装。 ### 生成配图 #### 覆盖率测试图（图10） 执行`./scripts/coverage.R` 生成的结果为一系列`coverage-*.pdf`格式文件 #### 不一致性分析图（图9） 执行`./scripts/consistency.R` 生成的结果为一系列`consistency-*.pdf`格式文件 #### 散点图（图8、图11） 执行`./scripts/scatter.R` 生成的结果为一系列`scatter-*.pdf`格式文件