Replication Package of the EMSE Journal Article: "In Search of Metrics to Guide Developer-Based Refactoring Recommendations"

This is a replication package and online appendix for the EMSE Journal paper "In Search of Metrics to Guide Developer-Based Refactoring Recommendations". Contents This repository contains the following: INSTALL: Detailed installation instructions for each of the used tools as well as the required Python dependencies. FIGURES: Graphical contents shared in the manuscript. Study design: Study design: Vertical diagram with the steps undergone during the study design. Data collection process diagram: Workflow diagram with the data collection process undergone. Sampling process diagram: Workflow diagram with the sampling process undergone during the data collection stage. Data analysis process diagram: Workflow diagram with the data analysis process undergone. CoT learning example: Example prompt from the performed prompt engineering stage based on the CoT learning technique. Results: RMC frequencies histogram: Histogram with the RMCs resulting from the performed Open-Coding. RF feature importance for process metrics: Results for process metrics based on the RF algorithm with MDA and MDG feature importance metrics. RF feature importance for product metrics: Results for product metrics based on the RF algorithm with MDA and MDG feature importance metrics. RF feature importance for all SW metrics: Results for all SW metrics based on the RF algorithm with MDA and MDG feature importance metrics. XGB feature importance for process metrics: Results for process metrics based on the XGB algorithm with the IG feature importance metric. XGB feature importance for product metrics: Results for product metrics based on the XGB algorithm with the IG feature importance metric. XGB feature importance for all SW metrics: Results for all SW metrics based on the XGB algorithm with the IG feature importance metric. Spearman's correlation matrix: Heatmap with the Spearman's correlation matrix with the performed uncorrected correlations, and the Bonferroni and BH-corrected correlations. Kendall's correlation matrix: Heatmap with the Kendall's correlation matrix with the performed uncorrected correlations, and the Bonferroni and BH-corrected correlations. TABLES: Tables appearing in the article. Study design: Study Context: Summary descriptive statistics of the projects considered in the study context. RMT detectable refactoring types: Refactorings Types detectable by RefactoringMiner 3.0 and newer versions. RMT mined output: Detected density of refactoring for each of the refactoring types detected by RefactoringMiner 3.0 and newer version. Computed Software metrics: Product and process metrics adopted in this study. Motivations: Motivations derived from developers’ answers based on their refactoring activity based on related work. Correlation analysis interpretability: Spearman’s and Kendall’s correlation coefficients' interpretation. refactoring-motivations-related: Definitions of the considered refactoring types with ground truth motivations. refactorings: Refactorings Types detectable by RefactoringMiner 3.0 and newer versions. Results: RQ1: LLMvsHumCT: LLM Motivation Alignment Agreement Contingency Table. Refactoring Type representativeness: Representativeness level for the refactoring types sampled in the manual validation for RQ$_1$. AgreementHvsLLM: Testing Human Motivation against LLM based motivation. Disagreement categories: Summary of manually detected disagreement categories between LLM extracted motivations and the literature. RQ2: ExtendedMotivations: Summary of LLM Motivations that Extend Human-Reported Motivations. HumanValidatedAlignment: Human Validated LLM Alignment with Literature. RQ3: Resulting RMCs from the Open-Coding stage: Description of the refactoring motivation categories (RMC) resulting from the Open Coding process that group the extracted refactoring motivations. CCR RMC: Refactoring Types Frequencies (#) for CCR RMC. CSRR RMC: Refactoring Types Frequencies (#) for CSRR RMC. Related Works: Related Work: Related work on the existing previous research on Refactoring Recommendation Appendix: Abbreviations: Table including all the refactoring types considered in this study with their abbreviations. Unique LLM-based refactoring motivations: List of unique motivation categories and their respective frequencies grouped into the defined RMCs. PROJECTS: List of the initial 124 Software projects considered in the study context. (Links to GitHub repositories provided). DATA: A folder containing all raw data extracted Compressed Software Repositories: Since the sampled refactoring observations were based on different versions of the adopted repositories, we cloned them and analyzed them at that specific stager of their development process. Providing the repositories made publishing this replication package impossible, therefore we provide a CSV table with the repository information to make the clonning of the specific version of the used repositories available. (IMPORTANT) NOTE: The real size of the folder containing the cloned software repositories summed up to 232GB. The table is size 58KB now. Mined output items: Similarly, the raw outputs resulting from RMiner's mining process as well as other additional mining outputs reached a total size of 178GB. We compressed the content into a 7z file ending up in 5GB approximately. The following itemized list presents the content within the compressed folder for one of the processed projects: Filtered RMT output: JSON file with the RMT output after cleaning mined commits without refactoring activity. Filtered RMT output (UTF-8): UTF-8 encoded JSON file with the RMT output after cleaning mined commits without refactoring activity. CoT input items: Pickle file with the content used in the prompts used in the CoT learning stage. Product and process metrics: JSON file with the computed product and process metrics. Version Control History data: JSON file with the ordered version control history data for the commits with detected refactoring activity. Version Control History data (UTF-8): UTF-8 encoded JSON file with the ordered version control history data for the commits with detected refactoring activity. Raw RMT output: Raw output from RMT with the detected refactoring activity. Class metrics: Computed class metrics with the CK tool. Method metrics: Additional computed method metrics with the CK tool (Not used, CK calculated them automatically). Delta between project size in prior work and this work: CSV table with the difference between the status of the analyzed projects from the prior work and the current work in terms of commits. Silva's Thematic Data: Ground-truth refactoring motivation data from Silva's adopted related work. Email password: Put your email password for remote emailing (https://developers.google.com/workspace/gmail/api/guides/sending) . GitHub tokens: Fill each line with a GitHub token, the script's logic will parse through the list automatically using all the request API rate. System Message Reasoning: Initial reasoning system message for the first interaction with Marco-o1 model. System Message Validation: Initial reasoning system message for the first interaction with Mistral NeMo, DeepSeek R1, Phi-4 models. RESULTS: Data files containing the analyzed results to answer the Research Questions. Raw results a folder with the results files for each of the sampling strategies. RQ1: Do motivations for refactoring in past studies align with those found in software project change histories? Prompt input data: JSON with input data per sampled observation for the CoT learning stage. LLM Validation results: JSON with the LLM validation results. 95% CI sample for validation: Drawn 95% CI sample with LLM validation results. Manually validated sample results: Manual validation on the 95% CI sample. RQ2: Are there additional motivations driving the developers’ willingness toperform refactoring? Prompt input data: JSON with input data per sampled observation for the CoT learning stage. LLM Validation results: JSON with the LLM validation results. 95% CI sample for validation: Drawn 95% CI sample with LLM validation results. Manually validated sample results: Manual validation on the 95% CI sample. RQ3: To what extent can product and process metrics capture the motivations driving the developers’ willingness to perform refactoring? Results based on LLM motivations: Prompt input data: JSON with input data per sampled observation for the CoT learning stage. LLM Validation results: JSON with the LLM validation results. 95% CI sample for validation: Drawn 95% CI sample with LLM validation results. Manually validated sample results: Manual validation on the 95% CI sample. Results based on Silva's motivations: Prompt input data: JSON with input data per sampled observation for the CoT learning stage. LLM Validation results: JSON with the LLM validation results. 95% CI sample for validation: Drawn 95% CI sample with LLM validation results. Manually validated sample results: Manual validation on the 95% CI sample. SCRIPTS: A folder containing all the scripts. Please, note that the numerical prefix denotes the order for launching the scripts. 00_clonning_repo_raw.py: Clones the projects, mines them with RefactoringMiner and collects version control history of refactoring commits. 01_stream_population_calculation.py: Calculates the population per refactoring type, refactoring commit and projects after the mining stage. 02_create_config.py: Calculates the sample size to reach statistical significance. 03_stream_sampling.py: Refactoring case sampling 04_stream_matching.py: Matching refactoring sample data with version control data. 05_stream_final_sample.py: Duplicate removal from obtained sample 06_ID_addition_to_refactorings.py: Since later on both LLM output and software metrics need to be merged, it adds to the input data an ID per item. 07_parallelized_motivation_cot_learning_2.py: Chain of Thought Few Shot Learning. 08_sample_repo_clonner.py: Clones every refactoring case repository at the commit level for later parallelization in during the metrics calculation. 09_per_item_metrics_calculation.py: Software metrics calculation 10_final_results_merge.py: Merges all the results from previous divided steps into single sample file with a refactoring observation per row. 11_reformat_silva_data_for_rq1.py: Reformats Silva's ground-truth data file to account for all refactoring operations, as the original file contained different refactorings included within the same commit as they were implemented as such within the same commit stage. 12_silva_cot_comparison_rq1.py: Performs CoT for comparison among the ground-truth motivations and the reasoned motivations on the same refactoring operation. 13_reformat_data_for_rq2.py: Reformats Silva's ground-truth data file to account for all refactoring operations, as the original file contained different refactorings included within the same commit as they were implemented as such within the same commit stage. It involves a different file format to deal with the CoT comparison to be implemented in the script No. 14. 14_silva_cot_comparison_rq2.py: Performs CoT for comparison among Silva's motivations and selected sample motivations on the representativeness level between the reported motivations and the registered commit message. 15_reformat_data_for_rq3.py: Reformats Silva's ground-truth data file to account for all refactoring operations, as the original file contained different refactorings included within the same commit as they were implemented as such within the same commit stage. It involves a different file format to deal with the CoT comparison to be implemented in the script No. 16. 16_llm_cot_comparison_rq3.py: Performs CoT learning for Open-Cfication over oding refactoring motivation categories in order to perform classithe motivation categories and the involved refactoring types (only involves ground-truth motivations). 16_silva_cot_comparison_rq3.py: Performs CoT learning for Open-Cfication over oding refactoring motivation categories in order to perform classithe motivation categories and the involved refactoring types (only involves selected sample motivations). 17_validation_sample_generator.py: Creates samples from the drawn motivation categories with the sample size drawn for the manual validation of the results. 18_commit_counter_data_collection.py: Simple script to obtain generic counts on the data handled during the study. Statistical Analysis: RQ1: R Script to execute analysis for RQ1 RQ3: R Script to execute analysis for RQ3 Summary statistics: R Script to execute analysis for the summary statistics for the adopted SW projects. Extra scripts: classes.py: Initial OO approach to mine the first insights of the projects. clonning_repo_yearly.py: Script containing the function to perform repository cloning and deletion. commons.py: Script containing common paths and functions reused during the entire project. get_sample_size.py: Simple script to get sample sizes with 95% of Confidence Interval. helpers.py: Script with help functions (Specially used during the first stages of the project) merge_data.py: Script to merge the initially collected data from RMiner. stream_population_calculation_project_test.py: Auxiliary script to calculate the population refactoring number from the concerning software projects. summary_statistics.py: Script calculating the summary statistics of the study context projects. License All generated data is provided under Creative Commons 4.0 Attribution License All scripts are provided under the MIT License All the analysed projects must be used in accordance with their respective licenses (shared in wach project when applicable). Preparation and installation Follow the instructions in INSTALL to install and configure all used tools. Mining Refactoring Activity with RefactoringMiner All the refactoring activity data is mined with the 3.0.10 version of the tool. The raw output for each PROJECT is copied to DATA_PATH/outputs/PROJECT_NAME/rminer_output.json. The execution of this process is performed in the script 00_clonning_repo_raw.py. This code works in a per project basis, as launching multiple mining processes with RefactoringMiner resulted into several processes reporting HEAP SPACE ERROR. In an attemp to speed up the processes, one would just add a for loop with the name of the projects. Similarly, the same script filters out the those analyzed commits that did not report any refactoring. The filtered version of the mined output is copied to DATA_PATH/outputs/PROJECT_NAME/filtered_rminer_output_utf8.json. We inform practitioners that collecting all the data from 119 projects out from the initial number of 128 took 2 months of computation with the used machine almost in an out of use state. Mining Version Control history with PyDriller Within the same initial script, 00_clonning_repo_raw.py, Pydriller collects the version control history from the analysed projects by inspecting the commits with refactoring activity, and further it orders the collection in a chronological format. The collected attributes are: Commit hash Commit message Commit date Commit author name Lines changed Files modified Boolean on whether it is a Refactoring commit or not Parent commit(s) And based on the boolean attribute, the filtering is performed. This version is copied to DATA_PATH/outputs/PROJECT_NAME/ordered_msg_rminer_output_utf8.json. Total refactoring activity calculation The script 01_stream_population_calculation.py parses all the mined refactoring output files and generates the JSON file in DATA_PATH/results/final_results.json with the following format. { "PROJECT_1_NAME": { "num_commits": int, "num_refactorings": int, "refactorings_per_type": { "REF_TYPE_1": int, "REF_TYPE_2": int, ... } }, "PROJECT_1_NAME": { ... } ... } Thus, we calculate the total refactoring activity population, per project as well as per refactoring type. Sampling strategies In this stage of the replication, first the sample size in order to reach statistically significant results is calculated. Then, the defined two sampling strategies are implemented. Getting the sample size The script 02_create_config.py script parses the already mentioned final_results.json and calculates the sample size with a 95% confidence interval and a margin of error of 5%. First sampling approach: The first sampling approach relies on the weighted sampling based on the mined number of refactoring output. The weights are computed in 02_create_config.py as well and can be found in DATA_PATH/results/sampled_cases.json. Secondly, the sampling is performed in script 03_stream_sampling.py in which based on the computed weights, refactoring cases from different projects are selected. The output can be found in DATA_PATH/results/sampled_refactorings.json. The script 04_stream_matching.py matches the data generated into sampled_refactorings.json with the version control history output stored in ordered_msg_rminer_output_utf8.json. Thus, the structure to perform the Chain of Thoughts Few-Shot Engineering stage receives as input refactoring cases with the JSON format as follows: [ { "refactoring_type": str, "project_name": str, "project_url": str, "hash": str, "file_path": str, "msg": str, "refactoring_description": str, "git_diff": str }, ... ] Additionally, since within the implementation there happened to be some duplicates, 05_stream_final_sample.py removes the duplicated items and reduces the sample size of the estimated statistically significant value. Second sampling approach: 3 cases per refactoring type & 3 cases per mined repository. The second sampling approach targets the collection of sample cases covering at least all the refactoring types, and with refactoring cases collected from all the analysed software projects. Such logic is implemented directly in the script 03_02_stream_sampling.py and the output is saved in DATA_PATH/results/sampled_refactorings_second_approach.json. Similarly as before The script 04_stream_matching.py matches the data generated into sampled_refactorings_second_approach.json with the version control history output stored in ordered_msg_rminer_output_utf8.json. Thus, the structure to perform the Chain of Thoughts Few-Shot Engineering stage receives as input refactoring cases with the JSON format specified before. Chain of Thoughts (CoT) Few-Shot-Learning (FSL) The script 06_parallelized_motivation_cot_learning_2.py performs the CoT FSL step. In order to automate this process, we used the LMStudio API, adapted to work with the Python library of OpenAI. The code as well as the prompting structure is based on the LMStudio's documentation in version 0.3.12. We provide instructions on how to set up all the requirements to successfully run this step in INSTALL. The implemented Large Language Models are: Reasoning model: Marco-o1 (7.62B parameters) NOTE: Initially offered by DevQuasar in HuggingFace, but no longer available Validation model 1: Mistral-Nemo-Base-2407 (12.2B parameters) Drop-in replacement of Mistral 7B Validation model 2: Phi-4 (14.7B parameters) Validation model 3: DeepSeek R1 Distill Qwen 14B (14.8B parameters) Distilled from DeepSeek's R1 reasoning model. Tuned for reasoning and chain-of-thought. The output from this script is collected in DATA_PATH/results/final_LLM_output.json.gz" if running the script for the first sample, or DATA_PATH/results/inal_LLM_output_second_approach.json.gz". Please, beware ro modify the input and output file paths in each script depending on the sampling strategy that is aimed to be replicated. NOTE: Once the script starts, it is expected to take approximately 2 days to finish. ID addition to the output Run the script 07_ID_addition_to_refactorings.py to add an ID to each of the obtained output items for the sample. This is just a matter of keeping all the items well identified. The output of this process will be stored in DATA_PATH/results/final_LLM_output_with_IDs.json.gz Repository clonning To make the Software metrics calculation parallelized and not run into race conflicts between nodes towards the same analyzed repository, we cloned every repository with the specified ID and performed git checkout to the refactoring commit where the refactoring was performed. Run script 08_sample_repo_clonner.py to launch this process. NOTE: This step might not be possible to perform in small machines given the size of some of the analyzed repositories. Software metrics calculation Run the script 09_per_item_metrics_calculation.py. It will take as input DATA_PATH/results/final_LLM_output_with_IDs.json and store the output with the metrics per item in DATA_PATH/results/final_metrics_results.json. NOTE: This calculation requires parsing through the entire version control history of the project up until the stage of the refactoring commit, per at least the number of times equal to the number of refactoring cases analyzed per each project. Merging the final results & CoT learning for each of the defined RQ Run script 10_final_results_merge.py in order to merge all the initial results into one CSV file. Run scripts from 11_reformat_silva_data_for_rq1.py up until 18_commit_counter_data_collection.py for performing CoT on the validation tasked to the employed LLMs for each of the RQs. Performing the statistical analysis Run summary-statistics.R to generate table with the summary statistics of the projects considered for this study. Run RQ1.R to perform the analysis to answer to our RQ1. Run RQ3.R to perform the analysis to answer to our RQ3. Product metrics COMM: The cumulative number of changes in a given file up to the considered commit. (PyDriller) ADEV: The cumulative number of active developers who modified a given file up to the considered commit. (PyDriller) DDEV: The cumulative number of distinct developers contributed to a given file up to the considered commit. (PyDriller) ADD: The normalized number of lines added to a given file in the considered commit. (PyDriller) DEL: The normalized number of lines removed from a given file in the considered commit. (PyDriller) OWN: The value indicates whether the file owner does the commit. (PyDriller + Git Blame) MINOR: The number of contributors who contributed less than 5% of a given file up to the considered commit. (PyDriller + Git Blame) SCTR: The number of packages modified by the committer in the considered commit. (PyDriller + Javalang) NADEV: The number of active developers who changed any of the files involved in the commits where the given file has been modified. (PyDriller) NDDEV: The number of distinct developers who changed any of the files involved in the commits where the given file has been modified. (PyDriller) NCOMM: The number of commits where the given has been involved. (PyDriller) NSCTR: The number of different packages touched by the developer in commits where the file has been modified. (PyDriller + Javalang) OEXP: The percentage of code lines authored by a given developer in the project. (Git) EXP: The mean of the experience of all developers across the project. (Git) Process metrics ND: The number of directories involved in a commit. (PyDriller) NS: Number of modified subsystems. (PyDriller) NF: Number of modified files. (PyDriller) ENTROPY: The distribution of the modified code across each given file in the considered commit. (Git+Math) LA: Ten lines added to the given file in the considered commit (absolute number of the ADD metric). (PyDriller) LD: The number of lines removed from the given file in the considered commit (absolute number of the DEL metric). (PyDriller) LT: The number of lines of code in the given file in the considered commit before the change. (PyDriller) FIX: Whether or not the change is a defect fix. (PyDriller) NDEV: The number of developers that changed the modified files. (PyDriller + Git Blame) AGE: The average period between the last and the current change. (Git) NUC: The number of times the file has been modified up to considered commit. (PyDriller) CEXP: The number of commits performed on the given file by the committer up to the considered commit. (PyDriller) REXP: The number of commits performed on the given file by the committer in the last month. (PyDriller) SEXP: The number of commits a given developer performs in the considered package containing the given file. (PyDriller + Javalang) Class metrics CBO: Coupling Between Object classes: measures the dependencies a class has. (CK) WMC: Weighted Methods per Class: sums the cyclomatic complexity of the methods in a class. (CK) RFC: Response For a Class: the number of methods in a class plus the number of remote methods that are called recursively. (CK) ELOC: Effective Lines Of Code: the lines of code excluding blank lines and comments. (SCC) NOM: Number Of Methods in a class. (CK) NOPM: Number Of Public Methods in a class. (CK) DIT: Depth of Inheritance Tree: the length of the path from a class to its farthest ancestor. (CK) NOC: Number Of Children (direct subclasses) of a class. (CK) NOF: Number Of Fields declared in a class. (CK) NOSF: Number Of Static Fields declared in a class. (CK) NOPF: Number Of Public Fields declared in a class. (CK) NOSM: Number Of Static Methods in a class. (CK) NOSI: Number Of Static Invocations of a class. (CK) HsLCOM: Henderson-Sellers revised Lack of Cohesion Of Methods (LCOM): a class cohesion metric based on sharing local instance variables by the class methods. (CK) ComRead: Comprehensive readability model: combines structural, visual (e.g., alignment), and textual features (e.g., comments readability). (CoRed)