Dataset for: Investigating the Perception and Effects of Misunderstandings in Java Code

Dataset and scripts used for thesis: Investigating the Perception and Effects of Misunderstandings in Java Code. # Contents of this archive The data and scripts used for the Master thesis `Investigating the Perception and Effects of Misunderstandings in Java Code`, by Chris Langhout. Provided for reproducibility and openness of data. In this readme, I try to specify what files are used for what, and what files include what results. ## folders ### /atoms-java Includes all .java files for the code examples used in the online experiment. ### /atoms_screenshots-java-normal Includes screenshots of all .java files. Generated by `autoScreenshot.py`. ### /data-analysis #### /data-analysis/atom-data - `java-answers.csv` set of computed results of snippets. Created by `javaAnswers.py`. - `new-atom-java.csv` contains atom names, in combination with their hashed counterparts. Used for combining answers back to their original questions, and generated by `autoScreenshot.py`. #### /data-analysis/survey-data Contains the export of the survey tool. Only in scanned `.ods` format. #### /data-analysis/grader.ts Used to grade the responses, and generate new datasets that are used for data analysis. Made use of the raw export of the survey response (not included for participant data protection, contact chris.langhout *at* gmail *dot* com for any other info). ### /result-data Output folder of `grader.ts` script. - `perceptAnswers.html` is used to find out what explanations were added by participants on the perception part of the experiment. To give context, the screenshots are used. Below each atom, the set of explanations is shown, based on what answer people gave. For better readability, the custom ordering this participant saw, is also included. e.g. (2 is ob) means that the right side screenshot the participant saw was the obfuscated variant. - `atomTimes.ods` includes a spreadsheet tab `RAWresultTable` which combines the counts of occurances, correct answers, incorrect answers, answers that were corrected by the grader, and total wrong answers. For every wrong answer, the given answer is displayed. Futhermore, the counts on how much participans were certain of their answer are counted and shown, split by correct/incorrect answer. An additional column with the comments placed here is added. The textual columns split different answers by a '|' character. - `effectOddsCI.ods` contains the same RAWresultTable, and also includes an `oddsRations` worksheet. Here the calculations for the odds ratios, including confidence intervals are shown. The `sheet3` highlights interesting atoms where the indicated confidence on answers is standing out. (this one does not work in excel, due to regular expression support. I used LibreOffice for this) - `perceptResultCounts.ods` shows the resulting counts of the perception side of the experiment. The sum of people that voted for the respective one of 4 options is displayed per atom. Additionally several data files, that are used to manually create one of the files mentioned above. ### /figures includes several `.xslx` files used for creating tables and figures. ## Scripts ### autoScreenshot.py used to open the individual .java files for the code examples in vscode. Have vscode open on the right half of the screen to make a screenshot, including the line numbers, excluding the first commented out lines, and save with the appropriate file name in the specified folder. Includes support for hashing the filenames, so that inspecting the images in the online experiment will not show the name of the atom of confusion. ### runJava.sh Create a `temp.java` file, copy the input into this file. Then try to compile this java code, run the compiled code (which prints results) and remove the created files again. save results in atom-data/java-answers.csv ### javaAnswers.py run the `runJava.sh` script over the code snippets and save the results in `java-answers.csv` ### gen_skeleton.py Make use of the data from https://atomsofconfusion.com/2016-snippet-study/questions.csv to generate files, and save metadata of the atoms-of-confusion from that source. <br> <br>本数据集与脚本用于学位论文：《探究Java代码中误解的感知与影响（Investigating the Perception and Effects of Misunderstandings in Java Code）》。 本归档文件内容如下：本归档包含Chris Langhout所撰硕士论文《探究Java代码中误解的感知与影响》所用的数据与脚本，旨在保障研究数据的可复现性与数据开放性。本自述文件将明确各文件的用途及对应包含的结果内容。 ### 文件夹 #### /atoms-java 包含在线实验中所用代码示例的全部Java文件。 #### /atoms_screenshots-java-normal 包含全部Java文件的截图，由`autoScreenshot.py`生成。 #### /data-analysis ##### /data-analysis/atom-data - `java-answers.csv`：代码片段的计算结果集，由`javaAnswers.py`生成。 - `new-atom-java.csv`：包含代码原子及其哈希后的对应项，用于将作答结果回溯至原始问题，由`autoScreenshot.py`生成。 ##### /data-analysis/survey-data 包含调研工具的导出文件，仅以扫描版.ods（Open Document Format）格式存储。 ##### /data-analysis/grader.ts 用于对调研作答结果进行评分，并生成用于数据分析的新数据集。该脚本使用了调研作答的原始导出文件（因涉及参与者隐私保护，未随包发布；如需获取更多信息，请联系[e-mail]）。 #### /result-data `grader.ts`脚本的输出文件夹： - `perceptAnswers.html`：用于查看参与者在实验感知环节所添加的解释内容，结合截图为作答提供上下文。每个代码原子下方会展示基于参与者作答对应的解释集合；为提升可读性，还会包含参与者所见的自定义作答顺序。例如：`(2 is ob)` 表示参与者所见的右侧截图为混淆变体（ob）。 - `atomTimes.ods`：包含名为`RAWresultTable`的电子表格标签，整合了作答次数、正确作答、错误作答、经评分人员修正的作答以及总错误作答的统计结果。针对每一处错误作答，均会展示具体作答内容。此外，该表格还统计并展示了参与者对自身作答的确定程度，按正确/错误作答的维度划分。本表格新增了注释列，文本列以`|`字符分隔不同作答。 - `effectOddsCI.ods`：包含与`atomTimes.ods`相同的`RAWresultTable`数据，并新增了`oddsRations`工作表，用于展示比值比（odds ratios）及其置信区间的计算结果。`sheet3`工作表标记了作答置信度表现突出的代码原子。因正则表达式支持问题，该工作表无法在Excel中正常加载，本研究使用LibreOffice打开该文件。 - `perceptResultCounts.ods`：展示了实验感知环节的结果统计，按每个代码原子统计了4个选项的投票人数总和。此外还包含若干用于手动生成上述提及文件所需的辅助数据文件。 #### /figures 包含用于制作表格与图表的若干.xlsx文件。 ### 脚本 #### autoScreenshot.py 用于在Visual Studio Code（VS Code）中打开单个代码示例Java文件。需将VS Code置于屏幕右半屏，以截取包含行号、排除首行注释的截图，并将文件以合适的文件名保存至指定文件夹。该脚本支持对文件名进行哈希处理，避免在线实验中查看图片时暴露代码误解原子（atoms-of-confusion）的名称。 #### runJava.sh 会创建临时文件`temp.java`，将输入内容复制至该文件，随后尝试编译该Java代码并运行（运行后会打印结果），最后删除生成的临时文件，并将结果保存至`atom-data/java-answers.csv。 #### javaAnswers.py 会对所有代码片段运行`runJava.sh`脚本，并将结果保存至`java-answers.csv`。 #### gen_skeleton.py 基于`https://atomsofconfusion.com/2016-snippet-study/questions.csv`中的数据生成文件，并保存该来源的代码误解原子（atoms-of-confusion）的元数据。