Replication Package for 'How do Machine Learning Models Change?'

Replication Package: How Do Machine Learning Models Change? Overview This replication package accompanies the paper "How Do Machine Learning Models Change?". In this study, we conducted a large-scale analysis of over 680,000 commits from 100,000 models and 2,251 releases from 202 of these models on the Hugging Face (HF) platform. Our goal was to understand how machine learning (ML) models evolve by classifying commit types using a detailed ML change taxonomy and analyzing temporal patterns in their activities using Bayesian networks. Our research addresses three main aspects: 1. Categorization of Commit Changes: We classified over 960,000 commits on HF, providing a detailed breakdown of change types and their distribution. 2. Analysis of Commit Sequences: We examined the sequence and dependencies of commit types using Bayesian networks to identify temporal patterns. 3. Release Analysis**: We investigated the distribution and evolution of release types, analyzing how model attributes and metadata change across successive releases. This package provides all the necessary code, data, and documentation to reproduce the results presented in our paper. Data Collection and Preprocessing Data Collection We collected data from the Hugging Face platform using the Hugging Face Hub API. The data extraction was performed up to May 2025, capturing details from over 1 million models available at that time. - Model & Release Information: We collected model metadata, commit histories, and release information (Git tags) for our sampled models. - Detailed Commit Changes: To get a detailed list of files modified in each commit, we implemented a direct Git processing approach. For each model, its repository was temporarily cloned to programmatically extract the list of changed files for every commit SHA. Data Preprocessing Commit Diffs We computed the differences for key JSON configuration files (e.g., `config.json`) between commits to identify added, deleted, and updated keys, which served as input for classification. Commit Classification We classified each commit according to Bhatia et al.'s ML change taxonomy using the Gemini 2.5 Flash LLM. To ensure the reliability of this process, we implemented a rigorous two-phase validation: 1. Prompt Refinement (Training): The prompt was iteratively refined over 6 cycles using a curated training set of 143 commits. The process was guided by comparing LLM classifications against a gold standard created by two human annotators (Human-Human IRR on a subset: 𝜅 = 0.7798). The final refined prompt achieved a Kappa of 0.9068 against the training gold standard. 2. Final Validation (Testing): The validated prompt was tested on an independent, statistically significant sample of 384 commits. The LLM's classifications achieved a Cohen's Kappa of 0.8568 when compared against the test set's gold standard, which itself was validated with a human-human IRR of 𝜅 = 0.8150. We also classified commits into Swanson's categories using a fine-tuned DistilBERT model, as detailed in the paper. Folder Structure The replication package is organized as follows. The structure has been designed to separate code, data, and metadata for clarity. `code/`: Contains all Jupyter notebooks for the study. `Collection/`: Scripts for data extraction from Hugging Face. `HFExtraction.ipynb`: Collects primary model and commit information. `HFReleasesExtraction.ipynb`: Collects release (tag) specific information. `Preprocessing/`: Scripts for data cleaning, processing, and classification. `HFCommitsPreprocessing.ipynb`: Processes commits, computes diffs, and prepares data for classification and analysis. `HFReleasesPreprocessing.ipynb`: Processes and classifies release data. `Analysis/`: Notebooks for reproducing the analysis for each research question. `HFFileChanges.ipynb`: Contains the preliminary analysis of file change patterns. `RQ1_Analysis.ipynb`: Analysis for Research Question 1. `RQ2_Analysis.ipynb`: Analysis for Research Question 2. `RQ3_Analysis.ipynb`: Analysis for Research Question 3. `datasets/`: Contains the key final datasets used in the analysis notebooks. `commits_datasets/`: Contains the main classified commit dataset. `HFCommitsClassification_final.csv`: The final dataset with over 960,000 classified commits for RQ1 and RQ2. `releases_datasets/`: Contains the datasets related to releases. `HFReleasesClassification.csv`: The final dataset of 2,251 classified releases for RQ3. `model_metadata.csv`: The extracted internal metadata from model files for RQ3.4. `metadata/`: Contains configuration files and the data used for the validation process. `validation_data/`: A sub-folder containing the gold standard data. `Agreement TOSEM Commit Changes.xlsx`: Excel containing details of the classication and validation processes. `prompt_refinement.txt`: The final, validated prompt used for the LLM classification along its previous iterations. `training_set_ground_truth.json`: Gold standard for the 143-commit training set. `training_set_first_classification.json`: First annotator's labels for the training IRR subset. `training_set_second_classification.json`: Second annotator's labels for the training IRR subset. `test_set_ground_truth.json`: Gold standard for the 384-commit test set. `test_first_classification.json`: First annotator's labels for the training IRR subset. `test_set_second_classification.json`: Second annotator's labels for the test IRR subset. `tags_metadata.yaml`: Auxiliary metadata file used during preprocessing. `README.md`: This file. `requirements.txt`: Lists the required Python packages. *Note: Other intermediate CSV files are provided to facilitate re-running specific parts of the analysis without starting from scratch.* How to Use This Package Setup 1. Create and activate a virtual environment (recommended). ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 2. Install required packages. ```bash pip install -r requirements.txt ``` Running the Analysis The Jupyter notebooks in the `code/` directory are numbered and named to be run in a logical sequence: Collection -> Preprocessing -> Analysis. We recommend following this order. - To reproduce our findings directly, you can start with the notebooks in `code/Analysis/`. They are configured to load the final, processed datasets provided in the `datasets/` folder. - To re-run the entire pipeline, start with the notebooks in `code/Collection/`, followed by `code/Preprocessing/`. Please note that running the full data collection and classification pipeline is time-consuming and may require significant computational resources and appropriate API keys for the LLM. Key Datasets Provided - For RQ1 & RQ2: `datasets/commits_datasets/HFCommitsClassification_final.csv` (100,000 models for RQ1; filtered to 14,343 models for RQ2). - For RQ3.1-3.3:`datasets/releases_datasets/HFReleasesClassification.csv` (2,251 releases from 202 models). - For RQ3.4: `datasets/releases_datasets/model_metadata.csv` (from 28 models). Contact If you have any questions or encounter issues with this package, please contact the corresponding author. If you find our work useful, please consider citing our paper.