Dataset for: Investigating the Perception and Effects of Misunderstandings in Java Code

Dataset and scripts used for thesis: Investigating the Perception and Effects of Misunderstandings in Java Code.

# Contents of this archive

The data and scripts used for the Master thesis `Investigating the Perception and Effects of Misunderstandings in Java Code`, by Chris Langhout.

Provided for reproducibility and openness of data.

In this readme, I try to specify what files are used for what, and what files include what results.

## folders

### /atoms-java

Includes all .java files for the code examples used in the online experiment.

### /atoms_screenshots-java-normal

Includes screenshots of all .java files. Generated by `autoScreenshot.py`.

### /data-analysis

#### /data-analysis/atom-data

- `java-answers.csv` set of computed results of snippets. Created by `javaAnswers.py`.

- `new-atom-java.csv` contains atom names, in combination with their hashed counterparts. Used for combining answers back to their original questions, and generated by `autoScreenshot.py`.

#### /data-analysis/survey-data

Contains the export of the survey tool. Only in scanned `.ods` format.

#### /data-analysis/grader.ts

Used to grade the responses, and generate new datasets that are used for data analysis. Made use of the raw export of the survey response (not included for participant data protection, contact chris.langhout *at* gmail *dot* com for any other info).

### /result-data

Output folder of `grader.ts` script.

- `perceptAnswers.html` is used to find out what explanations were added by participants on the perception part of the experiment. To give context, the screenshots are used.

Below each atom, the set of explanations is shown, based on what answer people gave. For better readability, the custom ordering this participant saw, is also included. e.g. (2 is ob) means that the right side screenshot the participant saw was the obfuscated variant.

- `atomTimes.ods` includes a spreadsheet tab `RAWresultTable` which combines the counts of occurances, correct answers, incorrect answers, answers that were corrected by the grader, and total wrong answers. For every wrong answer, the given answer is displayed.

Futhermore, the counts on how much participans were certain of their answer are counted and shown, split by correct/incorrect answer. An additional column with the comments placed here is added.

The textual columns split different answers by a '|' character.

- `effectOddsCI.ods` contains the same RAWresultTable, and also includes an `oddsRations` worksheet. Here the calculations for the odds ratios, including confidence intervals are shown. The `sheet3` highlights interesting atoms where the indicated confidence on answers is standing out. (this one does not work in excel, due to regular expression support. I used LibreOffice for this)

- `perceptResultCounts.ods` shows the resulting counts of the perception side of the experiment. The sum of people that voted for the respective one of 4 options is displayed per atom.

Additionally several data files, that are used to manually create one of the files mentioned above.

### /figures

includes several `.xslx` files used for creating tables and figures.

## Scripts

### autoScreenshot.py

used to open the individual .java files for the code examples in vscode.

Have vscode open on the right half of the screen to make a screenshot, including the line numbers, excluding the first commented out lines, and save with the appropriate file name in the specified folder.

Includes support for hashing the filenames, so that inspecting the images in the online experiment will not show the name of the atom of confusion.

### runJava.sh

Create a `temp.java` file, copy the input into this file. Then try to compile this java code, run the compiled code (which prints results) and remove the created files again. save results in atom-data/java-answers.csv

### javaAnswers.py

run the `runJava.sh` script over the code snippets and save the results in `java-answers.csv`

### gen_skeleton.py

Make use of the data from https://atomsofconfusion.com/2016-snippet-study/questions.csv to generate files, and save metadata of the atoms-of-confusion from that source.