Replication Package for What Do We Even Mean by Architectural Degradation? An Empirical Study of Proxy Metric (Dis)Agreement in Java projects

This repository contains the data-preparation and analysis code for the study: What Do We Even Mean by Architectural Degradation? An Empirical Study of Proxy Metric (Dis)Agreement in Java Projects Author information is omitted for double-anonymous review. NOTE: Before further reading this instructions file, beware that the hyperlinks pointing to the included repository files DO ONLY WORK once the repository is download and not directly in the preview format. The study investigates whether software metrics used as proxies for architectural degradation agree with one another, whether their evolution is consistent with directions reported in the literature, and how often those proxies provide contradictory signals during software evolution. The study uses the SQL release of the Software Quality Dataset (SQuaD). The scripts select Java projects, extract measurements produced by several tools, construct analysis-ready datasets at system, class, and method granularity, and run the statistical analyses for the three research questions. Contents The repository is organized below in workflow order. The extraction folders in steps 2--7 are independent and may be executed in any order after the Java project filter has been created. List_of_degradation_metrics.xlsx contains the complete list of metrics and the selection of architectural degradation proxy metrics available in the SQuaD dataset, along with their direction (increase or decrease) with architectural degradation and the list of papers supporting their selection. filtering_scripts/ contains the population-selection and dataset-construction workflow: repositories.txt is the initial list of repositories in owner#repository form; check_java_repositories.py checks the language composition with SCC; create_java_repo_list.py creates java-repositories.txt, the selected Java-project list; java_project_db_filter creates the corresponding database filter; java_projects_missing_release_data.txt records selected projects for which release data was unavailable; compute_source_system_metrics.py computes release-level commit, developer, bug, and modified-file measures; system_level_metrics/ contains alternative Python and SQL builders for the final system-level table; class_level_metrics/ filters, matches, and aggregates CK and Understand class measurements; and method_level_metrics/ filters, matches, and aggregates CK and JaSoMe method measurements. ck/ contains the CK class- and method-level extractors; find_limits.sql reports the available CK date range. jasome/ contains the JaSoMe class-, method-, and package-level extractors. process_metrics/ contains the release-level LOC and churn extractor. rminer/ contains the RMiner-based modified-files extractor. sonarqube/ contains the SonarQube method- and system-level extractors. understand/ contains the stored-procedure-based extractor that combines compatible Understand class tables. input_data/ contains the analysis-ready datasets used as inputs to the RQ1, RQ2, and RQ3 analyses at system, class, and method granularity. output_data/ contains the CSV outputs generated by the RQ1, RQ2, and RQ3 analysis scripts at system, class, and method granularit Analysis code & figures/ contains the RQ1, RQ2, and RQ3 R scripts and the retained PDF figures at system, class, and method granularity. This is the last stage of the workflow; its internal execution order is documented in Running the data analysis. Tables/ contains the supplementary tables used in the study: summary-statistics-msr-dataset.csv contains the descriptive statistics of the studied projects. correlation-interpretation.csv contains the thresholds used to interpret the strength of the correlation coefficients (e.g., weak, moderate, and strong) reported in the results. Research questions RQ1: To what extent are architectural degradation proxy metrics correlated in the direction expected from the literature? RQ2: How do architectural degradation proxy metrics evolve during the software evolution process? RQ3: Can a per-release/per-project disagreement index be constructed, and is it non-trivial and non-random across the corpus? Metrics The expected direction indicates how an increase in a metric is interpreted with respect to architectural degradation. Proxy Repository field Source Granularity Expected direction Coupling between objects cbo Understand Class Increase Lack of cohesion lcom Understand Class Increase Fan-in class_fanin, method_fanin CK Class, method Increase Fan-out class_fanout, method_fanout CK Class, method Increase Structural coupling complexity structural_coupling_complexity JaSoMe Method Increase Code churn churn Process/Git System Increase Code smells measures_code_smells SonarQube System Increase Development cost measures_development_cost SonarQube System Increase Lines of code total_LOC Process/SCC System Increase Reliability rating measures_software_quality_reliability_rating SonarQube System Decrease Technical debt measures_software_quality_maintainability_remediation_effort SonarQube System Increase Bugs num_bugs GitHub/Jira/Bugzilla System Increase Changes num_commits Git System Increase Developers num_developers Git System Increase Modified files modified_files Git/RMiner System Increase Execution order The complete workflow is: configure access to the SQuaD database; select the Java repositories and create the database filter; run the tool-specific extraction scripts; compute the source-system metrics; build the system-level table; build the class- and method-level tables and their release aggregates; export the analysis-ready database tables; and run the R scripts under Analysis code & figures/. According to the study design, the initial language selection identified 326 Java projects. SonarQube covered 324 of them and was used as the system-level anchor. Requiring complete values for all selected system-level proxies resulted in 16,427 releases from 314 projects for the final analysis. The scripts reflect the database state and recovery steps used during the study rather than a single automated pipeline. Check the prerequisites at the top of each SQL script before execution. Requirements Database A MySQL-compatible server is required. The scripts primarily use: SQuaD for CK, JaSoMe, SonarQube, process, release, and RMiner data; SQuaD_Und for Understand data; and SQuaD_Analysis for working and analysis-ready tables created by this package. The database user must be able to read the source schemas and create, drop, insert into, and index tables in SQuaD_Analysis. The Understand extraction also requires permission to create and execute a stored procedure. Software See INSTALL.md for platform setup, Python and R package installation, MySQL configuration, and verification commands. Python 3.10 or later; a MySQL client and server compatible with the included SQL; Git; SCC, available as scc on PATH; Python packages mysql-connector-python, numpy, pandas, PyDriller, and tqdm; and R packages tidyverse, nortest, trend, data.table, and ggplot2. Several scripts contain study-machine absolute paths. Use their command-line arguments where available or update the path constants for the local setup. Running the data preparation The commands below assume execution from the repository root. In the examples, $MYSQL denotes a configured MySQL client command with access to the three schemas described above. Stage 1: Select Java repositories Run the selection files in this order: cd filtering_scripts python3 check_java_repositories.py \ --repositories repositories.txt \ --output java_repository_results.csv python3 create_java_repo_list.py cd .. $MYSQL < filtering_scripts/java_project_db_filter The project list embedded in java_project_db_filter is not generated from java-repositories.txt; keep the two lists synchronized if the study population changes. Stage 2: Extract tool-specific measurements After SQuaD_Analysis.java_project_filter exists, run the extractors below. There are no ordering dependencies between the source-tool groups. # CK $MYSQL < ck/ck_class_metrics.sql $MYSQL < ck/ck_method_metrics.sql # JaSoMe $MYSQL < jasome/jasome_class_metrics.sql $MYSQL < jasome/jasome_method_metrics.sql $MYSQL < jasome/jasome_package_metrics.sql # Process metrics $MYSQL < process_metrics/process_metrics_selected.sql # RMiner $MYSQL < rminer/rminer_modified_files_java.sql # SonarQube $MYSQL < sonarqube/sonarqube_method_metrics.sql $MYSQL < sonarqube/sonarqube_system_metrics.sql # Understand $MYSQL < understand/extract_understand_class_metrics.sql These scripts create tool-specific tables in SQuaD_Analysis; most replace an existing table with the same name. Stage 3: Compute source-system metrics Run: DB_CONFIG=/path/to/db-config.env python3 filtering_scripts/compute_source_system_metrics.py \ --db-config "$DB_CONFIG" Use the script's command-line options to specify the release, commit, issue, project, and working paths for the local environment. The resulting table is SQuaD_Analysis.source_system_metrics_java. Stage 4: Build the system-level dataset Choose one of the following implementations. They are alternatives and both recreate SQuaD_Analysis.final_system_level_table_java. The Python implementation uses SonarQube releases as its anchor, resolves release-name variants, reuses available process measures, and reconstructs missing LOC and churn where possible: DB_CONFIG=/path/to/db-config.env RELEASE_DATA=/path/to/release-data.csv python3 filtering_scripts/system_level_metrics/build_final_system_level_table_java.py \ --db-config "$DB_CONFIG" \ --release-data "$RELEASE_DATA" The SQL implementation anchors on source_system_metrics_java and performs trimmed project/release joins without reconstructing missing LOC or churn: $MYSQL < filtering_scripts/system_level_metrics/build_final_system_level_table_java.sql Do not run the second implementation after the first unless replacing the table is intentional. Stage 5: Build the class-level dataset Run the numbered files in order: $MYSQL < filtering_scripts/class_level_metrics/01_filter_ck_class.sql $MYSQL < filtering_scripts/class_level_metrics/02_filter_understand_class.sql $MYSQL < filtering_scripts/class_level_metrics/03_join_class_level_metrics.sql $MYSQL < filtering_scripts/class_level_metrics/04_aggregate_class_metrics_by_release.sql 03_join_class_level_metrics.sql begins at the Understand-normalization stage and expects SQuaD_Analysis.ck_class_final_scope to exist. If matching is interrupted after both normalized scopes have been completed, resume with 03_resume_join_class_level_metrics.sql instead of rerunning the main step 03. The principal outputs are: SQuaD_Analysis.FINAL_CLASS_LEVEL_TABLE_JAVA_COMPLETE SQuaD_Analysis.FINAL_CLASS_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED The release aggregate contains the minimum, maximum, median, and mode of CBO, LCOM, fan-in, and fan-out. Stage 6: Build the method-level dataset Run the numbered files in order: $MYSQL < filtering_scripts/method_level_metrics/01_prepare_method_sources.sql $MYSQL < filtering_scripts/method_level_metrics/02_match_ck_jasome_methods.sql $MYSQL < filtering_scripts/method_level_metrics/03_create_final_method_table.sql $MYSQL < filtering_scripts/method_level_metrics/04_aggregate_method_metrics_to_release.sql The matcher progresses from exact normalized class/method/arity matches to unique relaxed matches. If it is interrupted after matching levels 1 and 2, run 02_resume_level3_ck_jasome_methods.sql in place of rerunning step 02. The principal outputs are: SQuaD_Analysis.FINAL_METHOD_LEVEL_TABLE_JAVA_COMPLETE SQuaD_Analysis.FINAL_METHOD_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED The release aggregate contains the minimum, maximum, median, and mode of method fan-in, method fan-out, and structural coupling complexity. Analysis-ready outputs Granularity Entity-level table Release-level table System FINAL_SYSTEM_LEVEL_TABLE_JAVA_COMPLETE One row per project/release Class FINAL_CLASS_LEVEL_TABLE_JAVA_COMPLETE FINAL_CLASS_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED Method FINAL_METHOD_LEVEL_TABLE_JAVA_COMPLETE FINAL_METHOD_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED The class- and method-level final-table scripts use FINAL_SYSTEM_LEVEL_TABLE_JAVA_COMPLETE as their release anchor. The included system builders create final_system_level_table_java; the subsequent step that creates the uppercase complete system table is not present in this checkout. The system builders also expect process_metrics_selected_java and sonarqube_system_metrics_java, while the extractors create the corresponding tables without the _java suffix. Some release-scope and normalized class staging tables are likewise prerequisites rather than products of the current scripts. Restore or recreate these study-database tables before running their dependents. Datasets The repository includes the analysis-ready input datasets and the output data generated by the RQ1--RQ3 analysis scripts input_data/ This directory contains the input datasets used by the system-, class-, and method-level analyses. final_analysis_project_list.csvList of projects included in the final analysis. FINAL_SYSTEM_LEVEL_TABLE_JAVA_COMPLETE.csvSystem-level dataset containing the architectural degradation proxy metrics collected for each project release. This dataset is used for the system-level analyses in RQ1, RQ2, and RQ3. FINAL_CLASS_LEVEL_TABLE_JAVA_COMPLETE.csvComplete class-level dataset containing individual class-level metric observations. It is used for analyses requiring class-level observations within project-release snapshots. FINAL_CLASS_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED.csvClass-level dataset aggregated by project and release. It contains distributional summaries used for longitudinal evolution and consecutive-release agreement analyses. FINAL_METHOD_LEVEL_TABLE_JAVA_COMPLETE.csvComplete method-level dataset containing individual method-level metric observations. It is used for analyses requiring method-level observations within project-release snapshots. FINAL_METHOD_LEVEL_TABLE_JAVA_RELEASE_AGGREGATED.csvMethod-level dataset aggregated by project and release. It contains distributional summaries used for longitudinal evolution and consecutive-release agreement analyses. output_data/ This directory contains the CSV output files generated by the analysis scripts for RQ1, RQ2, and RQ3 at the system, class, and method levels. Rq1classoutput/Contains the output CSV files generated from the RQ1 class-level analysis. Rq1methodoutput/Contains the output CSV files generated from the RQ1 method-level analysis. Rq1systemoutput/Contains the output CSV files generated from the RQ1 system-level analysis. Rq2classoutput/Contains the output CSV files generated from the RQ2 class-level analysis. Rq2methodoutput/Contains the output CSV files generated from the RQ2 method-level analysis. Rq2systemoutput/Contains the output CSV files generated from the RQ2 system-level analysis. Rq3classoutput/Contains the output CSV files generated from the RQ3 class-level analysis. Rq3methodoutput/Contains the output CSV files generated from the RQ3 method-level analysis. Rq3systemoutput/Contains the output CSV files generated from the RQ3 system-level analysis. Running the data analysis Analysis code & figures/ is the final repository folder in the execution order. Export the appropriate analysis-ready table, then update each R script's input/output path declarations for the local layout. The nine subfolders are independent after their inputs exist. Within each subfolder, always run the analysis script before its plotting script. RQ1: Correlation and expected-direction agreement Run the files in this order: cd "Analysis code & figures/RQ1System" Rscript Rq1system.r Rscript boxplotrq1system.r cd ../RQ1Class Rscript RQ1class.r Rscript Rq1classplot.r cd ../RQ1Method Rscript rq1method.r Rscript rq1methodplot.r cd ../.. The analysis scripts compute within-project correlations, select Pearson or Spearman correlation according to normality, and adjust p-values with the Benjamini-Hochberg method. The plotting scripts summarize median correlation, expected-direction agreement, statistically supported agreement, and the correlation-coefficient distributions. Each RQ1 folder retains the four corresponding PDF figures. RQ2: Metric evolution Run the files in this order: cd "Analysis code & figures/RQ2System" Rscript Rq2system.r Rscript lineplot.r cd ../RQ2Class Rscript Rq2class.r Rscript lineplot.r cd ../RQ2Method Rscript rq2method.r Rscript lineplotrq2.r cd ../.. The analysis scripts construct chronological project trajectories and apply the Mann-Kendall test and Sen's slope. System analysis uses the complete system table; class and method analysis use their release-aggregated tables. The plotting scripts create the normalized-evolution figures retained in each RQ2 folder. RQ3: Proxy disagreement Run the files in this order: cd "Analysis code & figures/RQ3System" Rscript Rq3system.r Rscript Rq3plot.r cd ../RQ3Class Rscript Rq3class.r Rscript Rq3classplot.r cd ../RQ3Method Rscript rq3method.r Rscript plotrq3method.r cd ../.. The analysis scripts transform changes between consecutive releases into directional signals, compute pairwise agreement statistics including Gwet's AC1, and calculate release- and project-level disagreement summaries. The plotting scripts create the agreement and disagreement PDF figures retained in each RQ3 folder. Reproducibility notes Run the SQL only in a dedicated working schema because many scripts replace output and intermediate tables. Resume scripts depend on completed intermediate state; read their header comments before using them. The Python and SQL system builders use different anchors and are not equivalent. Repository identifiers use owner#repository, for example apache#accumulo. Citation Citation information is omitted for double-anonymous review and will be added to the camera-ready version. License Attribution 4.0 International MIT License