Replication package of the paper "Where is Code Generated by LLMs Coming From? A Study with Gemini and Bing CoPilot"

# Replication Package This replication package contains the necessary tools, data, and scripts for reproducing the results of our paper: "*Where is Code Generated by LLMs Coming From? A Study with Gemini and Bing Copilot*". Below is a detailed description of the directory structure and the contents of this package. ## Contents The replication package is organized into two main directories: - `assets`: This directory contains all .csv files used as input for the script and the outputted .csv file used to perform the manual and automated analyses for RQ1 and RQ2. - `script`: This directory contains all scripts for RQ1 and RQ2. In the following, we describe the content of each directory: ## `assets` This directory contains the tools and resources required for our study. ### `dataset`: Contains the main datasets used in the study. - `annotationStore.csv`: Input dataset for our analyses, originating from the *CODESEARCHNET* dataset. - `queries.csv`: .csv file containing the queries used for the experiments filtered from the *CODESEARCHNET* dataset. This file contains the following columns: - *Language*: Programming language of the query - *Query*: Query used for the experiment - *GitHubUrl*: GitHub URL related to a snippet that addresses the query - *Relevance*: Relevance of the linked GitHub snippet to the query ### `data`: Contains the datasets and results of all analyses. - `queries.csv`: General input queries. This file contains the following columns: - *Language*: Programming language of the query - *Query*: Query used for the snippet generation - *Prompt*: LLM prompt generated for the query as: *You are a Senior ```<Language>``` developer. Then give me a ```<Language>``` code snippet about: ```<Query>```* - `queries_filled.csv`: Similar to the previous file, but also containing the output produced by the LLM-based assistants. This file contains the following columns: - *Language*: Programming language of the query - *Query*: Query used for the snippet generation - *Prompt*: LLM prompt generated for the query as: *You are a Senior ```<Language>``` developer. Then give me a ```<Language>``` code snippet about: ```<Query>```* - *Notes*: General notes that provide additional context or information about the query or prompt. - *Gemini_Answer(n)*: The generated code snippets by Gemini. - *Gemini(n)*: The external links provided by Gemini. - *Prompt (repeated)* - *Note*: Notes that provide additional context or information about the query or prompt. - *Copilot_Answer(n)*: The generated code snippets by Bing-Copilot. - *Copilot_Bing(n)*: The external links provided by Bing-Copilot. #### `copilot` || `gemini`: Contains the data related to the specific LLM. These two subdirectories have the same internal structure. - `queries.csv`: The `queries_filled.csv` file, filtered for the specific LLM. - `queries_noTrivial.csv`: Contains only the queries with at least one nontrivial generated snippet. - `external_links.csv`: External links extracted from the LLMs output. - `external_links_filled.csv`: Snippets extracted from the external links. - *index*: Query ID - *source*: Snippet ID - *url*: Link URL - *note*: Notes that provide additional context or information about the query or prompt - *code(n)*: The n-th code snippet extracted from the source #### `manual_analysis`: Manual analysis results. - `manual_analysis.csv`: - *index*: Query ID - *query*: Query used for the snippet generation - *generated_snippet_(n)*: The n-th code snippet generated by the LLM-based assistant - *trivial_1*: Manual analysis of whether or not the snippet was trivial (validator 1) - *trivial_2*: Manual analysis of whether or not the snippet was trivial (validator 2) - *trivial_final*: Manual analysis of whether or not the snippet was trivial (final classification if there is a disagreement) - *source*: URL to analyze - *source_type_1*: Type of the source (validator 1) - *source_type_2*: Type of the source (validator 2) - *source_type_final*: Type of the source (final classification if there is a disagreement) - *related_to_query_1*: Relevance of the link to the query (validator 1) - *related_to_query_2*: Relevance of the link to the query (validator 2) - *related_to_query_final*: Relevance of the link to the query (final classification if there is a disagreement) - *related_to_snippets_1*: Relevance of the generated snippet to those in the link (validator 1) - *related_to_snippets_2*: Relevance of the generated snippet to those in the link (validator 2) - *related_to_snippets_final*: Relevance of the generated snippet to those in the link (final classification if there is a disagreement) - `manual_analysis_noTrivial.csv`: As in the previous file, but only the queries with at least one nontrivial generated code snippet. #### `clone_detector`: Output and intermediate files for clone detection with Copilot data. - `copilot_tokens || gemini_tokens`: Contains the output the tokenization of the generated code snippets and the code snippets extracted from the external links. - `merged_llm_ext_link.csv`: All possible pairs (Cartesian product) (code snippet extracted from the external links, generated code snippet). This file is the input of the clone detection tool. - *ID_query*: Query ID - *query*: Query used for the snippet generation - *language*: Programming language of the query - *generated_snippet*: The generated code snippet by the LLM-based assistant - *ID_gen_snippet*: The index of the generated code snippet - *LOC_gen_snippet*: The number of lines of code of the generated code snippet - *ID_source*: Source ID - *source*: Source URL - *source_snippet*: Code snippet extracted from the source - *ID_source_snippet*: ID of the code snippet extracted from the source - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source - *note*: Notes that provide additional context or information about the query or prompt - `clone_detection_output.csv`: Contains the clone detection results. - *ID_query*: The index of the query - *query*: Query used for the snippet generation - *language*: The programming language of the query - *generated_snippet*: The generated code snippet by the LLM-based assistant - *ID_gen_snippet*: The index of the generated code snippet - *LOC_gen_snippet*: The number of lines of code of the generated code snippet - *ID_source*: Source ID - *source*: Source URL - *source_snippet*: Code snippet extracted from the source - *ID_source_snippet*: ID of the code snippet extracted from the source - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source - *note*: Notes that provide additional context or information about the query or prompt - *clone_detected*: bBolean value that indicates whether a clone has been detected (1 = detected, 0 = not detected) - *cloning_ratio*: Ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source - *cloned_lines*: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source #### `cosine_sim`: Cosine similarity results. - `cosine_sim_output.csv`: Contains the cosine similarity results - *query_id*: Query ID - *snippet_id*:ID the generated code snippet - *source_id*: ID of the source - *source_snippet_id*: ID of the code snippet extracted from the source - *cosine_similarity*: The cosine similarity between the generated code snippet and the code snippet extracted from the source #### `quant_analysis`: Quantitative analysis results. - `topN_links_se.csv`: Contains the top-N links extracted from the search engine. - *id*: Query ID - *query*: The query - *url*: Link URL - `merged_clone_cosine.csv`: Contains the merged results of the clone detection and cosine similarity. - *ID_query*: Query ID - *query*: The query - *language*: The programming language of the query - *generated_snippet*: The generated code snippet by the LLM-based assistant - *ID_gen_snippet*: The ID of the generated code snippet - *LOC_gen_snippet*: The number of lines of code of the generated code snippet - *ID_source*: The index of the source - *source*: The source URL - *source_snippet*: The code snippet extracted from the source - *ID_source_snippet*: The index of the code snippet extracted from the source - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source - *note*: Notes that provide additional context or information about the query or prompt - *clone_detected*: Boolean value that indicates if a clone has been detected(1 = detected, 0 = not detected) - *cloning_ratio*: The ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source - *cloned_lines*: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source - *cosine_similarity*: The cosine similarity between the generated code snippet and the code snippet extracted from the source - `results`: Final analysis results. - `jaccard_analysis.csv`: Contains the results of the Jaccard analysis comparing the provided external links by the LLMs with the top-N links extracted from the corresponding search engine. - *id*: Query ID - *language*: The programming language of the query - *llm_link*: The external links provided by the LLM - *llm_link_size*: The number of external links provided by the LLM - *overlap_links*: The overlapping links between the LLM and the search engine - *overlap_size*: The number of overlapping links between the LLM and the search engine - *non_overlap_links*: The non-overlapping links between the LLM and the search engine - *union_size*: The size of the union set links between the LLM and the search engine - *jaccard*: The Jaccard similarity between the LLM and the search engine - `merged_analysis.csv`: Contains the merged results of the manual and quantitative analyses. - *id*: The index of the query - *query*: The query used for the experiment - *trivial_final(n)*: The final assignment for the triviality of the n-th generated code snippet - *source*: The URL of the source - *source_type_final*: The final assignment for the type of the source - *related_to_query_final*: The final assignment for the relevance of the generated code snippet to the query - *related_to_snippets_final*: The final assignment for the relevance of the generated code snippet to the source - *cloning_ratio*: The maximum cloning ratio between the generated code snippet and all the code snippets extracted from the source - *cosine_similarity*: The cosine similarity related to the snippets with maximum cloning ratio - `cccfindersw-configuration-files`: Contains additional configuration files for the CCFinderSW clone detection tool. The files are `javascript_comment.txt` and `javascript_reserved.txt`. They must be placed in the tool's `comment/` and `reserved/` directories. ### `appendix.tex`: The appendix of the paper containing: - *Table 1*: Number of links of different types provided by Gemini and Bing CoPilot ### `appendix.pdf`: The appendix of the paper in PDF format. ## `script` This directory contains our scripts (mostly Python, an R script and an Applescript) to preprocess data and run the clone detection analyses. - `1_dateset_filtering.py`: Script to filter the dataset. The input of this script is the `annotationStore.csv` file, and the output is the `queries.csv` file. - `2_prompt_generation.py`: Script to generate prompts for the LLM-based assistants. The input of this script is the `queries.csv` file, and the output is the `queries_filled.csv` file. - `3_gen_sheet_sources_extraction.py`: Script to split the external links provided by the LLM, one for each row. The input of this script is the `queries_filled.csv` file, and the output is the `external_link.csv` file. - `4_ext_link_snippet_extraction.py`: Script to extract the snippets from Web URLs. It only works for the most popular domains. The input of this script is the `external_links.csv` file, and the output is the `external_links_filled.csv` file. - `5_top_n_link_SearchEngine.py`: Script to perform top-N link search using the corresponding search engines (Google Search and Bing). The input of this script is the `queries_filled.csv` file. It executes the `browser_bot.scpt`. The output is the `topN_links_se.csv` file. - `browser_bot.scpt`: Script for browser automation (AppleScript). - `6_se_vs_llm.py`: Script to compare (using the Jaccard metric) the links returned by the corresponding search engines with those provided by the LLM-based assistants. The input of this script is the `external_links.csv` file and the `topN_links_se.csv` file. The output is the `jaccaard_analysis.csv` file. - `7_results_manual_analysis.py`: Script to extract results and statistical analyses performed on the manual analysis and reported in the tables in the paper. - `8_merge_gen_source_snippets.py`: This script takes as input: `{llm}/queries.csv` and `{llm}/external_link_filled.csv` to merge them and generates an expanded one, i.e., one in which we have on each line a snippet extracted from the source, this will be the input of our final script for clone detection. The output is the `merged_llm_ext_link.csv` file. - `9_clone_detection.py`: Script to perform clone detection. The input of this script is the `merged_llm_ext_link.csv` file, and the output is the `clone_detection_output.csv` file. - `10_cosine_sim_check.py`: Script to compute the code snippets' cosine similarity. The script takes as input the tokenized files from the `{llm}_tokens` directory. The output is the `cosine_sim_output.csv` file. - `11_merger_clone_cosine.py`: Script to merge clone detection and cosine similarity results. The input of this script is the `clone_detection_output.csv` and the `cosine_sim_output.csv` files, and the output is the `merged_clone_cosine.csv` file. - `12_merge_manual_quantitative_analysis.py`: Script to merge manual and quantitative analysis results. The inputs of this script are the `manual_analysis.csv` and the `merged_clone_cosine.csv` files, and the output is the `merged_analysis.csv` file. - `cloningGraph.R`: R script to generate the cloning graph. The input of this script is the `merged_analysis.csv` file.