Replication Package for the Paper: Towards Automated Identification of Violation Symptoms of Architecture Erosion

Abstract of this Study Architecture erosion has a detrimental effect on maintenance and evolution, as the implementation deviates from the intended architecture. To prevent this, development teams need to understand early enough the symptoms of erosion, and particularly violations of the intended architecture. One feasible way is through the automated identification of architecture violations from textual artifacts, and particularly code reviews. In this paper, we developed 15 machine learning-based and 4 deep learning-based classifiers with three pre-trained word embeddings to identify violation symptoms of architecture erosion from developer discussions in code reviews. Specifically, we looked at code review comments from four large open-source projects from the OpenStack (Nova and Neutron) and Qt (Qt Base and Qt Creator) communities. We then conducted a survey and semi-structured interviews to acquire feedback from the involved participants who discussed architecture violations in code reviews, to validate the usefulness of our trained classifiers. Moreover, we conducted additional comparative experiments by evaluating Large Language Model (LLM)-based classifiers, including GPT-4o, Qwen-2.5, and DeepSeek-R1. The results show that, for ML/DL-based classifiers, the SVM classifier based on word2vec pre-trained word embedding performs the best with an F1-score of 0.779. In most cases, classifiers with the fastText pre-trained word embedding model can achieve relatively good performance. Moreover, 200-dimensional pre-trained word embedding models outperform classifiers that use 100- and 300-dimensional models. For ML/DL-based classifiers, ensemble classifiers based on the majority voting strategy can enhance the classifier and outperforms the individual classifiers. The findings derived from the online survey and interviews conducted with the involved developers reveal that the violation symptoms identified by our approaches have practical value and can provide early warnings for impending architecture erosion. Furthermore, LLM-based classifiers consistently outperform traditional ML/DL models, with GPT-4o yielding the highest F1-score of 0.851, though ensemble strategies offered no further performance gains for ensemble LLM-based classifiers in our case. We investigate the automated identification of violation symptoms from code reviews using both traditional ML/DL and state-of-the-art LLM techniques. Our contributions include the automated approach for identifying violation symptoms in code reviews, a systematic comparison of ML/DL and LLM approaches, and practitioner-centered insights into their practical usefulness, ultimately contributing to better architectural conformance and sustainability in software systems. Structure of the Replication Package data.zip includes: (1) extracted features (extracted_features) as inputs of classifiers, that is, word vectors of the extracted violation symptoms based on the three pre-trained word embedding models (i.e., word2vec, fastText, GloVe). (2) word_embedding includes pre-trained word embedding models. Due to the large size, we listed the download URL Download_url.txt, and we used embedding_dim.py to change the dimensions of the fastText models. (3) Violation symptoms.xlsx and Randomly_selected_comments.xlsx represent the review comments labeled as violations and non-violations, respectively. scripts.zip includes the Python scripts used to run the experiments, including data preprocessing and classifier training. survey and interview.zip include the survey form, interview protocol and questions, and the template of customized emails that we sent to participants. Experiment Steps 1. Preprocessing and feature extraction: Run feature_extraction.py to conduct preprocessing and feature extraction after adjusting appropriate parameters. It includes five steps: (1) Tokenization (2) Noise Removal (3) Stop words Removal (4) Capitalization Conversion (5) Stemming. Feature selection methods: word2vec, fastText, and Glove. 2. Training classifiers: Run Classifiers_ML.py to train machine learning-based classifiers. Run Classifiers_DL_classifiers.py to train deep learning-based classifiers. Run LLM.py to generate LLM-based classifiers. Run LLM_performance.py to evaluate the performance of LLM-based classifiers. Run LLM_voting.py to conduct voting strategy for ensemble LLM-based classifiers. Machine learning algorithms: Support Vector Machine (SVM), Logistic Regression (LR), Decision Tree (DT), Bernoulli Naive Bayes (NB), and k-Nearest Neighbor (kNN). Deep learning algorithm: TextCNN. Large language models: GPT-4o, Qwen-2.5, DeepSeek-R1. Experiment Environment Required packages and their versions: torch==1.11.0 numpy==1.22.3 gensim==4.1.2 fasttext==0.9.2 pandas==1.4.1 torchtext==0.12.0 sklearn==0.0 scikit-learn==1.0.2 w2vembeddings==0.1.2 matplotlib==3.5.1 tqdm==4.62.3 nltk==3.7 openai==1.95.1