Replication Package for Evaluating LLMs for Detecting ADR Violations

This replication package provides how we evaluate LLMs for detecting ADR Violations. The packge leverages the dataset of ADR files from Buchgeher et al. (ORIGINAL_README.md) to prepare the project's codebase and selected ADR files for the analysis with Large Language Models, and the LLM Detection Process. Note: All the hyperlinks referring to the shared files only work in the local version (downloading the relpication package). The only version will not work as the browser does not find the files. License All generated data is provided under DATA_LICENSE Creative Commons 4.0 Attribution License. All scripts are provided under the Script_LICENSE MIT License. Contents This repository consists of the following files: README.md: The README file describes the project and how to run the scripts to get the data and do the analysis INSTALL.md: Installation and configurations instructions requirements.txt: Library requirements for the Python environment Figures jaccard_C_heatmap.pdf: Heatmap diagram of pairwise jaccard coefficients (C) jaccard_overall_heatmap.pdf: Heatmap diagram of pairwise jaccard coefficients (overall) llm_process.pdf: Diagram of the general LLM Processing Implementation Data_collection ADR-dataset: The original dataset files (renamed from repositories) .env: Environment variables for the bash environment ORIGINAL_README.md: The original README file of the dataset repository adr_all.csv: Data on all ADR files from the original dataset reported in a single CSV adr_selected_projects.csv: Data on ADR files only from projects selected based on TLoC and the number of commits adr_template.csv: Data on ADR files from the selected projects for which the template was identified in the original dataset project_stats.csv: Data on TLoC and number of commits of each project Scripts 01_data_collection 01_clone_repos.sh: Script to clone locally the repositories from the dataset 02_create_codebase.sh: Script to merge source code files from repositories into a single JSON file 03_project_stats.sh: Script to analyze TLoC and the amount of commits of the repositories 04_list_adr_files.py: Script to process the dataset of ADR files to list all files, select ADR files from projects based on TLoC and commits, as well as select ADR files based on the parsed template 05_create_adr.sh: Script to move the selected ADR files to the projects' codebases for analysis with LLMs 02_llms adr_judge.py: Script to using LRM to detect the ADR violations adr_validate.py: Script to using 3 LLMs to validate the ADR judgement results from LRM sbatch.sh: Script to submit the work to the supercomputer 03_data_analysis rq1analysis.py: Script to calculate the overall agreement (Fleiss' Kappa, Agreement by category, and pairwise Jaccard coefficients) rq2analysis.py: Script to calculate the sklearn metrics Results llms judge_marco.csv: Raw results generated from LRM (Marco-o1) judge_marco_statuscorrection.csv: Results through the status filtering status_category.csv: Status extraction and mapping validate_mistral.csv: Raw results (validation) generated from Mistral Nemo validate_llama.csv: Raw results (validation) generated from Llama validate_qwen.csv: Raw results (validation) generated from Qwen human_validation.csv: Human validation process for the sample size results RQ1 agreement_analysis.csv: Input data sent to the rq1analysis.py to calculate the overall agreement rq1data.csv: Generated data (Overall Inter-LLMs Agreement and Agreement by Label with Class Frequency) RQ2 performance_analysis.csv: Input data sent to the rq2analysis.py to calculate the sklearn metrics rq2data.csv: Generated data (table that present all the needed sklearn metrics) RQ3 rq3.1data.csv: Results for the ADR category (limitation analysis) rq3.2data.csv: Results for the error labels generated by LLMs (error label analysis) Replication of the results This section describes the steps necessary to replicate the project, ADR selection, and ADR violation detection using LLMs steps from our work. Setting up the environment Follow the instructions in INSTALL.md to set up the environment and define environment variables. Cloning the repositories Use the script 01_clone_repos.sh to attempt to clone all repositories from the dataset. The scripts clones all repositories to the directory given by the $REPO_DIR environment variable. Cloning is performed over https, and the repository is skipped if the URL is not found (project has been deleted) or if authentication is requested (project has been switched to private). For each repository, the owner and the project name are concatenated using -- to create the unique string identifying the project to be used as the directory, i.e. for the project URL https://github.com/OWNER/REPO the directory will be OWNER--REPO. This convention is used all throughout this repository and the scripts. Creating the codebase files Use the script 02_create_codebase.sh to merge the source code of the cloned projects into JSON files. The script goes through all the projects in the $REPO_DIR, identifies all the source code files by common extensions, checks if the files are text, non-empty and in UTF-8 encoding, and merges them into a single JSON file. The files are written to the codebase directory given by $CODEBASE_DIR environment variable for each project. Analyzing project statistics Use the script 03_project_stats.sh to compute the TLoC and the number of commits of each project. The script count the number of code lines from codebase files found in $CODEBASE_DIR and uses git to check the number of commits in $REPO_DIR. The data is saved to the CSV file given in $STATS_FILE environment variable. Selecting the ADR files Use the script 04_list_adr_files.py to parse the original dataset and select the ADR files. The script performs the following: Iterate over all files from the original ADR_dataset and write information on each repository owner, name and URL, ADR file, its parsed template and acceptance status into the CSV file adr_all.csv Select the cloned projects based on the TLoC (4th quartile) and number of commits (2nd quartile and above) given in project_stats.csv, and save data about only their ADR files to the CSV file adr_selected_projects.csv Further select only the ADR files that have their template parsed and save them to the CSV file adr_template.csv The file adr_template.csv will thus contain all ADR files selected for analysis from the original dataset. Adding ADR files to the codebase Use the script 05_create_adr.sh to add the selected ADR to the codebases. The script takes all ADR files given in adr_template.csv, attempts to find them in the corresponding cloned repository in $REPO_DIR and copy to the corresponding codebase in $CODEBASE_DIR. If a file is not found it is skipped. Final data selection structure After all scripts are successfully executed, the $CODEBASE_DIR has the following structure: $CODEBASE_DIR/ OWNER1--REPO1/ OWNER1--REPO1--codebase.json ADR1.md ADR2.md ... OWNER2--REPO2/ OWNER2--REPO2--codebase.json ADR1.md ADR2.md ... .../ Files in this structure can then be given to LLMs. LRM Running After we get the above data with the described structure, we use it as the input for running LRM (Marco-o1). We use adr_judge.py and sbatch.sh, submit the work to the supercomputer, with the following command structure: sbatch sbatch.sh "MODEL_NAME" PORT OUTPUT_FILE Then we get the results judge_marco.csv. Then we analyze the status situation of each project, try to map the status and remove the projects whose status is "rejected". Finally, we get the results judge_marco_statuscorrection.csv. Now the LRM Running process is completed. Three LLMs Validation Running After we get the results from LRM, we continue using 3 LLMs to validate the results. We use adr_validate.py and sbatch.sh, submit the work to the supercomputer with the same command structure. Then we get the validation results from 3 LLMs: validate_mistral.csv, validate_llama.csv, and validate_qwen.csv. Human Validation and Data Analysis (RQs) We selected the sample size for human validation. Then we do the data analysis based on the defined RQs. All the needed script, data, and results are shown in the "Scripts - data analysis" and "Results - RQ1/RQ2/RQ3" folders.