Using White-Box Test Generation During Development: A Replicated Study

This package contains the artifacts for our external, differentiated replication of an empirical study on using white-box test generation. The original experiment is described in the following paper. José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: a controlled experiment and think-aloud observations. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), 338–349. DOI: 10.1145/2771783.2771801 30 developers and students participated in the experiment in January 2016. The main difference in the replication was using the IntelliTest test generator tool instead of EvoSuite. The contents of the package are the followings. /assignments session_starts_endings.csv - Timestamps when each session was started assignments_with_projects.csv - Assigments of participants to projects, classes and sessions /projects /golden - Contains the golden implementation and test suites of all 4 classes /user - Contains the user implementation and test method stubs of all 4 classs /results time_merged.csv - File of timing analysis containing each test run of subjects results.csv - Contains the coverage and other required data of each participant tasks exit.csv - Answers of the exit survey background.csv - Answers of the background questionnaire /scripts /ps Get-Results.ps1 - Gets the results of both tasks for one specified participant (invokes MSBuild, MSTest, VisualMutator, OpenCover) Get-Failures-Errors.ps1 - Gets failures and errors of both tasks for one specified participant (invokes MSBuild, MSTest) Get-TimeData.ps1 - Iterates over all test runs for each participant /r likert.R - Generates the Likert chart si_st.R - Generates table of subject implementation executed with subject test suites gi_st.R - Generates table of results where subject test suites was run on golden implementation time_spent_tests.R - Generates a table of time and activity of testing errors_failures.R - Generates table of errors and failures obtained using the golden test suite correlation_analysis.R - Generates table containing correlation between lack of correctness and IntelliTest usage本套件包含我们针对一项白盒测试生成（white-box test generation）实证研究开展的外部差异化复现所用的全部成果。原始实验详见以下论文：José Miguel Rojas、Gordon Fraser与Andrea Arcuri，2015年，《软件开发过程中的自动化单元测试生成：一项受控实验与有声思维观测研究》，收录于2015年国际软件测试与分析研讨会（ISSTA 2015）论文集，第338–349页，DOI: 10.1145/2771783.2771801。2016年1月，共有30名开发者与学生参与了本次实验。本次复现的核心差异在于采用IntelliTest测试生成工具替代了原实验中的EvoSuite工具。 本套件的内容如下： 1. /assignments 目录 session_starts_endings.csv — 记录各实验时段的起始与结束时间戳 assignments_with_projects.csv — 记录被试与实验项目、类及实验时段的分配关系 2. /projects 目录 /golden — 包含全部4个类别的标准实现（golden implementation）与测试套件（test suite） /user — 包含全部4个类别的用户实现与测试方法桩代码（test method stubs） 3. /results 目录 time_merged.csv — 时序分析文件，收录所有被试的各次测试运行数据 results.csv — 收录所有被试任务的覆盖率及其他必要实验数据 exit.csv — 退出调查问卷的作答结果 background.csv — 背景调查问卷的作答结果 4. /scripts 目录 /ps 子目录： Get-Results.ps1 — 获取指定被试的两项任务结果（需调用MSBuild、MSTest、VisualMutator、OpenCover工具） Get-Failures-Errors.ps1 — 获取指定被试的两项任务的失败与错误信息（需调用MSBuild、MSTest工具） Get-TimeData.ps1 — 遍历所有被试的全部测试运行数据 /r 子目录： likert.R — 生成李克特（Likert）量表图表 si_st.R — 生成被试实现在其自身测试套件下的执行结果表格 gi_st.R — 生成被试测试套件在标准实现上运行的结果表格 time_spent_tests.R — 生成测试活动与耗时统计表 errors_failures.R — 生成使用标准测试套件得到的错误与失败统计表 correlation_analysis.R — 生成正确性缺失程度与IntelliTest使用情况之间相关性的统计表格