Artifact for "Does Task Complexity Moderate the Benefits of Liveness? - A Controlled Experiment"

This artifact includes the setup, data, and analysis for the experiment described in the article "Does Task Complexity Moderate the Benefits of Liveness? - A Controlled Experiment" published in the journal "The Art, Science, and Engineering of Programming" Volume 9.

The artifact has the following structure:

experiment-setup

experiment-environment.zip: Archive including the fully configured Squeak/Smalltalk environment

experiment-protocol.pdf: Full protocol for conducting the experiment

system: Includes the base system used for the experiment without any seeded faults

tasks

task-descriptions.txt: The task descriptions of steps to reproduce and symptoms for the tasks used in the experiment

patches: Patch files for all generated tasks

infrastructure: Contains source code of the tools used for controlling tasks in the development environment

questionnaire

survey.dfglive.2023-01-06.xml: Questionnaire configuration for a SoSci survey server, includes demographics, experience, and skill questionnaire

results

data

export-NN.zip: Each archive contains the raw data from one run

starTrackData-NN: Contains the detailed event log

experimentState.json: Contains the measurements and the final state for each task (completed, notStarted, etc.)

skillTestResult-NN.json: Contains the score reached in the skill test

task-X-NN.cs: The submitted patch for task X

questionnaire

data_dfglive-analysis.ods: Demographics and experience questionnaire results (gender column is redacted due to potential deanonymization)

results of skill test are only available in graded form in the participant export files

analysis

analysis-r: Main analysis scripts

main.R, main-contrast-based.R: Two versions of the main and moderation effect analysis using two different processes

live-tools-usage-time.R: Main and moderation effect analysis on tool usage

demographics.R: Analysis of demographics and rendering of charts

analysis-tool-usage-correlation: Includes scripts to extract tool usage frequencies from the event logs. Assumes that export-NN.zip files are in the parent folder.