Replication package: ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code

README This is a replication package/supplementary data package for the study ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code. This data package is intended to be used as a companion to the study paper itself. If you are interested in running a similar study, you will find the files we used to administer this study in 0-study_setup/ If you are interested in the raw data we collected for this study, we have shared anonymized information in 1-collected_data/ If you are interested in how we performed our analysis, you will find the relevant files in 2-data_analysis/ Contents . ├── 0-study_setup │ ├── demographics_survey-questions.pdf │ ├── experimenter_handbook.pdf │ ├── field_notes_template.pdf │ └── post_task_questionnaire-questions.pdf ├── 1-collected_data │ ├── chatgpt_logs │ │ ├── p10_chat.html │ │ ├── p10_conversations.json │ │ ├── p11_chat.html │ │ ├── p11_conversations.json │ │ ├── p2_chat.html │ │ ├── p2_conversations.json │ │ ├── p3_chat.html │ │ ├── p3_conversations.json │ │ ├── p7_chat.html │ │ └── p7_conversations.json │ ├── post_task_questionnaire-responses.csv │ ├── sketches │ │ ├── P1.svg │ │ ├── P10.svg │ │ ├── P11.svg │ │ ├── P2.svg │ │ ├── P3.svg │ │ ├── P4.svg │ │ ├── P5.svg │ │ ├── P6.svg │ │ ├── P7.svg │ │ └── P9.svg │ ├── task-starter-template-ai.zip │ └── task-starter-template-nonai.zip ├── 2-data_analysis │ ├── codebook.csv │ ├── scripts │ │ ├── analyze_vscode_logs.py │ │ └── polya_state_timeline.tgz │ └── vignettes.md └── README.md 7 directories, 32 files Explanation of files 0-study_setup demographics_survey-questions.pdf Reproduced list of questions asked to participants to screen them and collect general information before they booked an in-person study session experimenter_handbook.pdf Structured handbook for the scheduled physical study sessions that were arranged with participants after they completed their demographics survey. During the sessions, participants were required to bring their own device. They were also asked to come prepared with: (1) VS Code and Zoom already installed on their device, and (2) a charger for their laptop Note that although the participant study sessions had durations of three hours, most participants ran out of time to complete their task, and therefore needed to be stopped by the experimenter so they could move onto the post-task questionnaire. field_notes_template.pdf: Template used by the experimenter running the physical study sessions to jot notes while observing each participant complete their task. post_task_questionnaire-questions Reproduced list of questions that were administered to participants at the conclusion of their task, or when only 1-collected_data chatgpt_logs/ For all ChatGPT participants, there are two files: p#_chat.html - an HTML export of their ChatGPT conversation(s) p#_conversations.json - a JSON export of their ChatGPT conversation(s) All participants used the same ChatGPT account. Between each participant session the account data was exported, and all conversations and memories were cleared from the account. If a system prompt was created by the participant, this was also reset to the default. post_task_questionnaire-responses.csv The data collected from participants' post-task questionnaires NOTE: Participants were not allowed access to any resources (or codebase) when completing the post-task questionnaire. sketches/ Sketches drawn during the post-task questionnaire by participants, prompted by the wording: Sketch a component diagram listing all components you remember that were relevant in completing the task and their relationships (see post_task_questionnaire-questions.pdf) All sketches have been reproduced electronically to maintain participant confidentiality NOTE: Participants were not allowed access to any resources (or codebase) when sketching their component diagram. task-starter-template-ai.zip and task-starter-template-nonai.zip GitHub repos that was cloned by AI and nonAI participants respectively into GitHub Codespaces, using a general GitHub account to maintain participant confidentality, when they were to begin their task. 2-data_analysis codebook.csv The qualitative codebook used for this study. Codes are broken into two sections in the hierarchy: Polya codes (also known as "PS Codes") - used to identify the problem solving state of participants throughout their task Note the four codes (EMO, WHA, DOU, and DEL) in the "Uncategorized" section, which we occasionally applied during the coding process in combination with another PS Code. We did not use these in our analysis, but found them similarly useful as to the screen/modes codes to assist in indexing our data. Screen/modes codes - used to contextualize what participants are currently "doing" in their task Codes were applied by researchers manually in Excel, by specifying the start and end points of every interval in the screen recording where a qualitative code applied, and noting down the qualitative code for that section scripts/ analyze_vscode_logs.py - script used to analyze the VS Code logs we collected from participants polya_state_timeline.tgz - code used to generate the condensed state diagrams of Polya codes - created with Observable vignettes.md An extended collection of the vignettes that are included in our paper to narratively explain how participants found themselves stuck while completing their tasks