Replication Package: Block-based or Graph-Based? Why Not Both? Designing a Hybrid Programming Environment for End-users

Block-based or Graph-based? Why Not Both? Designing a Hybrid Programming Environment for End-users: Replication Package

This repository contains supplementary materials for the paper "Block-based or Graph-based? Why Not Both? Designing a Hybrid Programming Environment for End-users". We provide this data for transparency reasons and to support replications of our experiments.

Note: This package is anonymized for peer review purposes. We will provide contact information for the authors at a later date. We also plan to add interactive versions of our tasks and tutorials for an easier exploration of our study design.

Summary of files contained in this package

This package contains two parts:

The data-analysis/ folder contains the raw dataset we collected for our experiment in CSV format, as well as scripts we used for our analyses.

Column ID contains a unique 4-digit identifier for each participant that they were assigned throughout our study.

Column Group contains the group (Blocks/Graph) that participants were randomly assigned to.

Columns Task1Time and Task2Time contain the time participants spent to complete the two programming tasks of our study in minutes.

Columns Task1Success and Task2Success contain a boolean value indicating whether the participants successfully completed the given task. Note that participants had unlimited attempts until they timed out after a strict time limit of 30 minutes, so if a participant was unsuccessful the corresponding time value is 30.

Columns Task1Tests and Task2Tests contain the number of times a participant executed their code throughout a task, including their final submission if they were successful.

Columns LearnTask, ReadTask and WriteTask contain the scores that participants gave to the task editor component of their assigned programming environment. There are 3 scores for the categories "learnability", "readability" and "writability". Scores are on a 5-point scale from 1 (worst) to 5 (best).

Columns LearnTrig, ReadTrig and WriteTrig contain the scores that participants gave to the trigger editor component of their assigned programming environment. There are 3 scores for the categories "learnability", "readability" and "writability". Scores are on a 5-point scale from 1 (worst) to 5 (best).

Columns LearnComp, ReadComp and WriteComp contain the scores that participants gave to their assigned assigned programming environment in direct comparison to the other alternative. There are 3 scores for the categories "learnability", "readability" and "writability". Unlike in the paper, where scores are on a scale from -2 to 2, the raw scores here are on a 5-point scale from 1 (strong preference for other environment) to 5 (strong preference for own environment).

The script successplot.py was used to generate the success rate plot used in a figure in the paper

The script survival.py was used to perform the survival analysis presented in the paper and generate the related figure.

The script batplot.py was used to generate the 3x3 grid of ratings used in a figure in the paper.

The materials/ folder contains the tutorials and task descriptions we presented to study participants. It also contains the exact wording of pre-screening and post-experiemental survey questions.

The image pre-screening.png shows the three pre-screening questions we used to determine whether our participants could be included in our study.

The images tutorial1_instructions.png and tutorial1_sim.png contain the instructions and initial simulator state we provided to participants for the first programming tutorial. This tutorial did not provide starter code and was identical for both participant groups.

The images tutorial2_instructions.png and tutorial2_sim.png contain the instructions and initial simulator state we provided to participants for the second programming tutorial. This tutorial was identical for both participant groups and provided participants with starter code, which is shown in the images:

tutorial2_code_main.png for the main program in the left canvas

tutorial2_code_move.png for the definition of "Move box to the right".

The images tutorial3_instructions_blocks.png/tutorial3_instructions_graph.png and tutorial3_sim.png contain the instructions and initial simulator state we provided to participants for the third programming tutorial. This tutorial also provided participants with starter code, which is shown in the images:

tutorial3_code_main.png for the main program in the left canvas

tutorial3_code_pick.png for the definition of "Pick up box"

tutorial3_code_place.png for the definition of "Place box"

The images task1_instructions.png and task1_sim.png contain the instructions and initial simulator state we provided to participants for the first programming task. The task did not provide starter code and the instructions were identical for both participant groups.

The images task2_instructions.png and task2_sim.png contain the instructions and initial simulator state we provided to participants for the second programming task. The instructions were identical for both groups. This task also provided participants with starter code, which is shown in the images:

task2_code_main.png for the main program in the left canvas

task2_code_pick_prog.png for the definition of "Pick up block"

task2_code_load_trig_blocks.png/task2_code_load_trig_graph.png for the definition of the trigger "Ready to load machine"

task2_code_load_prog.png for the definition of "Load and activate machine"

task2_code_finished_trig_blocks.png/task2_code_finished_trig_graph.png for the definition of the trigger "Machine finished"

task2_code_finished_prog1.png for the definition of "Get block from machine"

task2_code_finished_prog2.png for the definition of "Place block in bin"

The document post_survey_full.pdf contains a raw export of the comprehension questions and post-experimental survey as they were presented to participants 

The image usability.png shows the usability questions we used to determine a participant's rating of their assigned programming environment. The questions were identical for both participant groups.

The images comprehension_blocks_1.png and comprehension_blocks_2.png show the program comprehension questions we used to determine whether participants in the Blocks group could understand more complex triggers.

The images comprehension_graph_1.png and comprehension_graph_2.png show the program comprehension questions we used to determine whether participants in the Graph group could understand more complex triggers.

The images comparison_blocks.png and comparison_graph.png show the images of triggers in the alternative environment that we showed to our participants before choosing their preferred environment. The questions were identical for both participant groups.

The image comparison.png shows the questions we used to determine a participant's preference between the two programming environment alternatives.