Lost at C: Data from the Security-focused User Study

<strong># 2022 Study on the security implications of Large Language Model Code Assistants</strong> This repository contains the results of the 2022 study described in the Paper: `Lost at C: A User Study on the security implications of Large Language Model Code Assistants` Link: https://arxiv.org/pdf/2208.09727.pdf <strong>This is the version 0.1 repository, which contains a subset of the data. For the full repository, please click to the version 0.2 within Zenodo.</strong> Here, the overall goal is to determine if users with access to code suggestions via a Large Language Model (OpenAI code-cushman-001) in a GitHub Copilot-like arrangement produce code with a higher incidence rate of security-related bugs than those without any such access. In particular we concern ourselves with low-level memory-related bugs such as those often present in buggy C code. To answer this question, a User Study was conducted (N=58) which had users implement a shopping list in C as a singly-linked list. Half the users had access to a custom Copilot-like extension which generated suggestions according to code-cushman-001, and half had no access or coding hints other than provided by Visual Studio Code's default Intellisense.<br> The study was performed in a controlled environment (a virtualized cloud-based desktop). This task was made deliberately difficult than usual via the specifications: participants had to implement the shopping list according to an unusual API containing a number of pitfalls. They had to implement only the implementation of the specification (i.e. the `list.c` file). Users were provided `list.h` as well as a suite of automated (if basic) tests. For more details, you can see the associated paper. The repository contains the following open information: <strong>./recombined_list_files/assisted/[UUID]-list.c</strong><br> - The set of files in this directory contain the implementations<br> of the shopping lists made by the users provided suggestions by<br> the large language model.<br> - Each UUID refers to an individual study participant.<br> - Each file contains additional annotations describing the bugs<br> identified during the security analysis. <strong>./recombined_list_files/control/[UUID]-list.c</strong><br> - The set of files in this directory contain the implementations<br> of the shopping lists made by the users who were not provided<br> suggestions by the large language model.<br> - Each UUID refers to an individual study participant.<br> - Each file contains additional annotations describing the bugs<br> identified during the security analysis. <strong>./user_instructions</strong><br> - This directory contains the starting point provided to each<br> participant, including instructions (in a README.md file), the<br> file needing to be implemented (list.c), and the other support files. <strong>./bug_finding_records.sqlite3</strong><br> - This SQLITE3 database contains anonymous participant details and<br> information on the bugs found in a convenient queryable format.<br> - Internally, tables:<br> `active_inactive`: ID and demographics per participant<br> `functions_marked`: detailing each function marked (per user)<br> `bug_classes`: each unique bug class found across all users<br> `bugs_found`: each unique bug ID matching<br> bug_classes to functions to users <strong>./suggestion_cover.html</strong><br> - This standalone HTML file shows the relationship between code<br> submitted by participants assisted by the large language model and<br> the suggestions provided by those models. You can see, for instance,<br> if an individual line of code was written by the user, written by<br> the model, or some combination thereof. <strong>./README.md</strong><br> - This document.<br>