# Blocks? Graphs? Why Not Both? Designing and Evaluating a Hybrid Programming Environment for End-users: Replication Package

<strong>Blocks? Graphs? Why Not Both? Designing and Evaluating a Hybrid Programming Environment for End-users: Replication Package</strong> This repository contains supplementary materials for the paper "Blocks? Graphs? Why Not Both? Designing and Evaluating a Hybrid Programming Environment for End-users". We provide this data for transparency reasons and to support replications of our experiemnts. <em>Note: This package is anonymized for peer review purposes. We will provide contact information for the authors at a later date. We also plan to add interactive versions of our tasks and tutorials in an updated version to allow readers easier exploration/experimentation.</em> <strong>Summary of files contained in this package</strong> This package contains two parts: The <code>data-analysis/</code> folder contains the raw dataset we collected for our experiment in CSV format, as well as scripts we used for our analyses. Column <code>ID</code> contains a unique 4-digit identifier for each participant that they were assigned throughout our study. Column <code>Group</code> contains the group (Blocks/Graph) that participants were randomly assigned to. Columns <code>Task1Time</code> and <code>Task2Time</code> contain the time participants spent to complete the two programming tasks of our study in minutes. Columns <code>Task1Success</code> and <code>Task2Success</code> contain a boolean value indicating whether the participants successfully completed the given task. Note that participants had unlimited attempts until they timed out after a strict time limit of 30 minutes, so if a participant was unsuccessful the corresponding time value is 30. Columns <code>Task1Tests</code> and <code>Task2Tests</code> contain the number of times a participant executed their code throughout a task, including their final submission if they were successful. Columns <code>LearnTask</code>, <code>ReadTask</code> and <code>WriteTask</code> contain the scores that participants gave to the task editor component of their assigned programming environment. There are 3 scores for the categories "learnability", "readability" and "writability". Scores are on a 5-point scale from 1 (worst) to 5 (best). Columns <code>LearnTrig</code>, <code>ReadTrig</code> and <code>WriteTrig</code> contain the scores that participants gave to the trigger editor component of their assigned programming environment. There are 3 scores for the categories "learnability", "readability" and "writability". Scores are on a 5-point scale from 1 (worst) to 5 (best). Columns <code>LearnComp</code>, <code>ReadComp</code> and <code>WriteComp</code> contain the scores that participants gave to their assigned assigned programming environment in direct comparison to the other alternative. There are 3 scores for the categories "learnability", "readability" and "writability". Unlike in the paper, where scores are on a scale from -2 to 2, the raw scores here are on a 5-point scale from 1 (strong preference for other environment) to 5 (strong preference for own environment). The script <code>survival.py</code> was used to perform the survival analysis presented in the paper and generate the related figure. The script <code>batplot.py</code> was used to generate the 3x3 grid of ratings used in a figure in the paper. The <code>materials/</code> folder contains the tutorials and task descriptions we presented to study participants. It also contains the exact wording of pre-screening and post-experiemental survey questions. The image <code>pre-screening.png</code> shows the three pre-screening questions we used to determine whether our participants could be included in our study. The images <code>tutorial1_instructions.png</code> and <code>tutorial1_sim.png</code> contain the instructions and initial simulator state we provided to participants for the first programming tutorial. This tutorial did not provide starter code and was identical for both participant groups. The images <code>tutorial2_instructions.png</code> and <code>tutorial2_sim.png</code> contain the instructions and initial simulator state we provided to participants for the second programming tutorial. This tutorial was identical for both participant groups and provided participants with starter code, which is shown in the images: <code>tutorial2_code_main.png</code> for the main program in the left canvas <code>tutorial2_code_move.png</code> for the definition of "Move box to the right". The images <code>tutorial3_instructions_blocks.png</code>/<code>tutorial3_instructions_graph.png</code> and <code>tutorial3_sim.png</code> contain the instructions and initial simulator state we provided to participants for the third programming tutorial. This tutorial also provided participants with starter code, which is shown in the images: <code>tutorial3_code_main.png</code> for the main program in the left canvas <code>tutorial3_code_pick.png</code> for the definition of "Pick up box" <code>tutorial3_code_place.png</code> for the definition of "Place box" The images <code>task1_instructions.png</code> and <code>task1_sim.png</code> contain the instructions and initial simulator state we provided to participants for the first programming task. The task did not provide starter code and the instructions were identical for both participant groups. The images <code>task2_instructions.png</code> and <code>task2_sim.png</code> contain the instructions and initial simulator state we provided to participants for the second programming task. The instructions were identical for both groups. This task also provided participants with starter code, which is shown in the images: <code>task2_code_main.png</code> for the main program in the left canvas <code>task2_code_pick_prog.png</code> for the definition of "Pick up block" <code>task2_code_load_trig_blocks.png</code>/<code>task2_code_load_trig_graph.png</code> for the definition of the trigger "Ready to load machine" <code>task2_code_load_prog.png</code> for the definition of "Load and activate machine" <code>task2_code_finished_trig_blocks.png</code>/<code>task2_code_finished_trig_graph.png</code> for the definition of the trigger "Machine finished" <code>task2_code_finished_prog1.png</code> for the definition of "Get block from machine" <code>task2_code_finished_prog2.png</code> for the definition of "Place block in bin" The image <code>usability.png</code> shows the usability questions we used to determine a participant's rating of their assigned programming environment. The questions were identical for both participant groups. The images <code>comprehension_blocks_1.png</code> and <code>comprehension_blocks_2.png</code> show the program comprehension questions we used to determine whether participants in the Blocks group could understand more complex triggers. The images <code>comprehension_graph_1.png</code> and <code>comprehension_graph_2.png</code> show the program comprehension questions we used to determine whether participants in the Graph group could understand more complex triggers. The images <code>comparison_blocks.png</code> and <code>comparison_graph.png</code> show the images of triggers in the alternative environment that we showed to our participants before choosing their preferred environment. The questions were identical for both participant groups. The image <code>comparison.png</code> shows the questions we used to determine a participant's preference between the two programming environment alternatives.