A Benchmark Dataset for Evaluating the Provenance of LLM-Generated Code

A Benchmark Dataset for Evaluating the Provenance of LLM-Generated Code

Introduction

This dataset has been created to evaluate the provenance of LLM-generated code snippets by analyzing their similarity to external sources. It accompanies our study on Gemini and Bing CoPilot. Below is a detailed description of the directory structure and the contents of this package.

Contents

The dataset is organized into two directories, each corresponding to one of the LLM-based assistants used in our study:

bingCopilot/: Contains datasets related to Bing CoPilot.

gemini/: Contains datasets related to Gemini.

The file structure and content are identical in both directories, differing only in the LLM that generated the data.

raw_data.csv

This file contains all raw data used to query the LLM-based assistants and their generated outputs. The columns are as follows:

id: Identifier of the query

language: Programming language of the query

query: Query used for the experiment (e.g., You are a Senior <Language> developer. Then give me a <Language> code snippet about: <Query>)

prompt: The prompt used to query the LLM-based assistant

note: Notes that provide additional context or information about the querying process

generated_snippet\_(n): The n-th code snippet generated by the LLM-based assistant

source_link\_(n): The n-th source link provided by the LLM-based assistant

triviality.csv

This file contains the triviality classification of the generated code snippets. It contains the following columns:

id: Identifier of the query

prompt: The prompt used to query the LLM-based assistant

generated_snippet\_(n): The n-th code snippet generated by the LLM-based assistant

trivial\_(n): Triviality classification of the n-th code snippet (yes = trivial, no = non-trivial)

extracted_snippets.csv

Snippets extracted from the external links. It contains the following columns:

id: Identifier of the query

source_id: Identifier of the external link

source_link: Link to the external source

note: Notes that provide additional context or information about the querying the LLM-based assistant

code(n): The n-th code snippet extracted from the source

manual_annotations.csv

It contains the manual annotations of the generated code snippets and the external links provided by the LLM-based assistants. It contains the following columns:

id: Identifier of the query

generated_snippet_(n): The n-th code snippet generated by the LLM-based assistant

trivial_(n): Annotation of whether the snippet was trivial (yes = trivial, no = non-trivial)

source_link: Link to the external source

source_link_type: Classifying the source type(e.g., Q&A, forum, blog, documentation, tutorial, etc.)

related_to_query: Relevance of the link to the query (yes = relevant, no = not relevant)

related_to_snippets: Relevance of the generated snippet to those in the link (yes = relevant, no = not relevant)

similarity_analysis.csv

This file aggregates the results of clone detection and cosine similarity analysis. It contains the following columns:

id: Identifier of the query

prompt: The prompt used to query the LLM-based assistant

generated_snippet_(n): The n-th code snippet generated by the LLM-based assistant

source_link: Link to the external source

max_cloning_ratio: Maximum percentage of lines in the generated snippet detected as clones in the extracted source code

max_cosine_similarity: Maximum cosine similarity between the generated code snippet and the code snippet extracted from the source