Using White-Box Test Generation During Development: A Replicated Study

This package contains the artifacts for our external, differentiated replication of an empirical study on using white-box test generation.

The original experiment is described in the following paper.

José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: a controlled experiment and think-aloud observations. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), 338–349. DOI: 10.1145/2771783.2771801

30 developers and students participated in the experiment in January 2016.

The main difference in the replication was using the IntelliTest test generator tool instead of EvoSuite.

The contents of the package are the followings.

/assignments

session_starts_endings.csv - Timestamps when each session was started

assignments_with_projects.csv - Assigments of participants to projects, classes and sessions

/projects

/golden - Contains the golden implementation and test suites of all 4 classes

/user - Contains the user implementation and test method stubs of all 4 classs

/results

time_merged.csv - File of timing analysis containing each test run of subjects

results.csv - Contains the coverage and other required data of each participant tasks

exit.csv - Answers of the exit survey

background.csv - Answers of the background questionnaire

/scripts

/ps

Get-Results.ps1 - Gets the results of both tasks for one specified participant (invokes MSBuild, MSTest, VisualMutator, OpenCover)

Get-Failures-Errors.ps1 - Gets failures and errors of both tasks for one specified participant (invokes MSBuild, MSTest)

Get-TimeData.ps1 - Iterates over all test runs for each participant

/r

likert.R - Generates the Likert chart

si_st.R - Generates table of subject implementation executed with subject test suites

gi_st.R - Generates table of results where subject test suites was run on golden implementation

time_spent_tests.R - Generates a table of time and activity of testing

errors_failures.R - Generates table of errors and failures obtained using the golden test suite

correlation_analysis.R - Generates table containing correlation between lack of correctness and IntelliTest usage