Replication Pacakge for Architectural Changes in the Wild: Can We Catch 'Em All?

Replication Package This package contains the datasets, source code, validation, and generated outputs for Architectural Changes in the Wild: Can We Catch 'Em All? The study addresses: RQ1: How prevalent are documented architecturally significant changes across software development histories? RQ2: What architectural concerns characterize documented architecturally significant changes? The project selection contains 449 repositories from SQuaD dataset. Note: All hyperlinks referring to files in this replication package are intended to be used after downloading the package. Local file links may not work when browsing the package online. License All generated data is provided under DATA_LICENSE Creative Commons 4.0 Attribution License. All scripts are provided under SCRIPT_LICENSE MIT License. Package contents Paths below are relative to the package root, Replicationpackage_Architecture_Changes_in_the_wild. Keep spelling and capitalization when running commands. Path Contents and purpose projects.csv Project list; Collection scripts use owner, repo_name, and, for commits, html_url. Codeforgettingcommitsdata/getcommitmsgsdata.py Clones repositories and extracts hashes, author metadata, dates, subjects, and bodies using git log --all. Codeforgettingcommitsdata/commit_messages_data_by_project/ Collected commit CSVs organized by repository. Codeforgettingpullrequestdata/getpullrequestdata.py Collects PR data through the GitHub GraphQL API with pagination and checkpointing. Codeforgettingpullrequestdata/pull_requests_data_by_project/ Collected PR CSVs organized by repository. splittingintoprojects.py Pre-classification splitting utility. Its execution arguments must be checked against this file. Llm_analysis_code/01_commit_architecture_rag.py RAG classification client for commits. The classification prompt is embedded in the script. Llm_analysis_code/02_pull_request_architecture_rag.py RAG classification client for PR titles and bodies. The classification prompt is embedded in the script. Llm_analysis_code/slurm.sh Roihu commit job launcher, vLLM startup, and checkpoint continuation. Llm_analysis_code/slurm_pull_requests.sh Corresponding Roihu PR launcher. AnalysisRQ1commit/repository_results_after_LLM/ Per-repository classified commit data used by RQ1. AnalysisRQ1commit/rq1_commit.py Complete commit-level RQ1 analysis. AnalysisRQ1commit/resultrepositorysep.py Splits an exported classified commit archive by rag_source_file; this is a different stage from splitting raw collected commits. AnalysisRQ1commit/complete_result/ Saved commit-level RQ1 summaries and figures. AnalysisRQ1pull/pull_request_repository_results/ Per-repository classified PR data used by RQ1. AnalysisRQ1pull/rq1_pull_request_analysis.py Complete PR-level RQ1 analysis. AnalysisRQ1pull/complete_pr_result/ Saved PR-level RQ1 summaries and figures. mapping releases/map_all_projects.py Release-mapping utility. mapping releases/release_data.csv Release metadata supplied for the study. mapping releases/results_commit_mapping/ Saved commit mapping outputs. mapping releases/results_pr_pr_mapping/ Saved PR mapping outputs. AnalysisRQ2/rq2.py Concept normalization, frequencies, co-occurrences, release-level analysis, and figures for commits or PRs. AnalysisRQ2/RQ2_commit_result/ Saved commit-level RQ2 outputs. AnalysisRQ2/RQ2_pr/ Saved PR-level RQ2 outputs. human_validation_results_calculations.xlsx Sampled classifications, human-review entries, and calculation sheets. DATA_LICENSE, SCRIPT_LICENSE Licensing information distributed with the package. README.md Package overview, contents, and replication workflow. INSTALL.md Environment setup and execution instructions. requirements.txt Required packages. Data Files File Description Codeforgettingcommitsdata/commit_messages_data_by_project/ Collected commit messages and metadata, stored in one CSV per repository. Codeforgettingpullrequestdata/pull_requests_data_by_project/ Collected pull-request titles, descriptions, and metadata, stored in one CSV per repository. projects.csv Repository list used for data collection. mapping releases/release_data.csv Release metadata. human_validation_results_calculations.xlsx Sampled model outputs, human-review entries, and calculations. Validation Workbook Sheet Name Description Contents Overview of the workbook's sheets and their contents. Calculation Summary calculations for the commit-message and pull-request validation samples. human_validation_commit Sampled commit-message classifications from Marco, Llama, Mistral, and Qwen, together with supporting outputs and human-review entries. human_validation_PR Sampled pull-request classifications from the four models, together with supporting outputs and human-review entries. Replication of the Results First follow INSTALL.md to install dependencies and configure paths. Python dependencies are listed in requirements.txt. The primary model was ATH-MaaS/Marco-o1, executed through vLLM on the CSC Roihu supercomputer. Retrieval used sentence-transformers/all-MiniLM-L6-v2 and SWEBOK Guide v4, Chapter 2. Validation materials also contain outputs from Llama 3.1 8B Instruct, Mistral NeMo Instruct 2407, and Qwen2.5 14B Instruct. 1. Data Collection Use getcommitmsgsdata.py in Codeforgettingcommitsdata/ and getpullrequestdata.py in Codeforgettingpullrequestdata/, with projects.csv. The PR collector requires a GitHub read token. Prepare raw commit CSVs by repository using splittingintoprojects.py. 2. LLM Classification Use Llm_analysis_code/01_commit_architecture_rag.py for commits and 02_pull_request_architecture_rag.py for PRs. On Roihu, submit slurm.sh or slurm_pull_requests.sh; each launcher starts vLLM and its classification client. Configure the input paths, CSC allocation, and Chapter-2-only reference PDF before submission. 3. Validation The sampled classifications, model comparisons, and human-review entries are provided in human_validation_results_calculations.xlsx. Each artifact type commit messages and pull request description has 385 reviewed records. 4. Data Preparation and Release Mapping Organize classified outputs by repository. AnalysisRQ1commit/resultrepositorysep.py splits a classified commit export. Release-mapping resources and saved outputs are in mapping releases/, including map_all_projects.py, results_commit_mapping/, and results_pr_pr_mapping/. 5. RQ1 Run AnalysisRQ1commit/rq1_commit.py using repository_results_after_LLM/, and AnalysisRQ1pull/rq1_pull_request_analysis.py using pull_request_repository_results/. These produce summary CSVs and figures. 6. RQ2 Run AnalysisRQ2/rq2.py separately with --kind commit and --kind pr, using the release-mapped CSVs and local Git clones. Outputs include Excel workbooks, UpSet and release-evolution figures, and Sankey text. Installation Detailed commands for all execution stages are in INSTALL.md.