Replication Package for 'How do Machine Learning Models Change?'

Replication Package: How Do Machine Learning Models Change? Overview This replication package accompanies the paper "How Do Machine Learning Models Change?". In this study, we conducted a large-scale analysis of over 680,000 commits from 100,000 models and 2,251 releases from 202 of these models on the Hugging Face (HF) platform. Our goal was to understand how machine learning (ML) models evolve by classifying commit types using a detailed ML change taxonomy and analyzing temporal patterns in their activities using Bayesian networks. Our research addresses three main aspects: 1. Categorization of Commit Changes: We classified over 960,000 commits on HF, providing a detailed breakdown of change types and their distribution. 2. Analysis of Commit Sequences: We examined the sequence and dependencies of commit types using Bayesian networks to identify temporal patterns. 3. Release Analysis**: We investigated the distribution and evolution of release types, analyzing how model attributes and metadata change across successive releases. This package provides all the necessary code, data, and documentation to reproduce the results presented in our paper. Data Collection and Preprocessing Data Collection We collected data from the Hugging Face platform using the Hugging Face Hub API. The data extraction was performed up to May 2025, capturing details from over 1 million models available at that time. - Model & Release Information: We collected model metadata, commit histories, and release information (Git tags) for our sampled models. - Detailed Commit Changes: To get a detailed list of files modified in each commit, we implemented a direct Git processing approach. For each model, its repository was temporarily cloned to programmatically extract the list of changed files for every commit SHA. Data Preprocessing Commit Diffs We computed the differences for key JSON configuration files (e.g., `config.json`) between commits to identify added, deleted, and updated keys, which served as input for classification. Commit Classification We classified each commit according to Bhatia et al.'s ML change taxonomy using the Gemini 2.5 Flash LLM. To ensure the reliability of this process, we implemented a rigorous two-phase validation: 1. Prompt Refinement (Training): The prompt was iteratively refined over 6 cycles using a curated training set of 143 commits. The process was guided by comparing LLM classifications against a gold standard created by two human annotators (Human-Human IRR on a subset: 𝜅 = 0.7798). The final refined prompt achieved a Kappa of 0.9068 against the training gold standard. 2. Final Validation (Testing): The validated prompt was tested on an independent, statistically significant sample of 384 commits. The LLM's classifications achieved a Cohen's Kappa of 0.8568 when compared against the test set's gold standard, which itself was validated with a human-human IRR of 𝜅 = 0.8150. We also classified commits into Swanson's categories using a fine-tuned DistilBERT model, as detailed in the paper. Folder Structure The replication package is organized as follows. The structure has been designed to separate code, data, and metadata for clarity. - `code/`: Contains all Jupyter notebooks for the study. - `Collection/`: Scripts for data extraction from Hugging Face. - `HFExtraction.ipynb`: Collects primary model and commit information. - `HFReleasesExtraction.ipynb`: Collects release (tag) specific information. - `Preprocessing/`: Scripts for data cleaning, processing, and classification. - `HFCommitsPreprocessing.ipynb`: Processes commits, computes diffs, and prepares data for classification and analysis. - `HFReleasesPreprocessing.ipynb`: Processes and classifies release data. - `SwansonsClassification.ipynb`: Classifies commits into corrective, perfective, and adaptive types. - `Analysis/`: Notebooks for reproducing the analysis for each research question. - `HFFileChanges.ipynb`: Contains the preliminary analysis of file change patterns. - `RQ1_Analysis.ipynb`: Analysis for Research Question 1. - `RQ2_Analysis.ipynb`: Analysis for Research Question 2. - `RQ3_Analysis.ipynb`: Analysis for Research Question 3. - `datasets/`: Contains the key final datasets used in the analysis notebooks. - `commits_datasets/`: Contains the main classified commit dataset. - `HFCommitsClassification_final.csv`: The final dataset with over 960,000 classified commits for RQ1 and RQ2. - `releases_datasets/`: Contains the datasets related to releases. - `HFReleasesClassification.csv`: The final dataset of 2,251 classified releases for RQ3. - `model_metadata.csv`: The extracted internal metadata from model files for RQ3.4. - *Note: Other intermediate CSV files are provided to facilitate re-running specific parts of the analysis without starting from scratch.* - `metadata/`: Contains configuration files and the data used for the validation process. - `validation_data/`: A sub-folder containing the gold standard data. - `Agreement TOSEM Commit Changes.xlsx`: Excel containing details of the classication and validation processes. - `prompt_refinement.txt`: The final, validated prompt used for the LLM classification along its previous iterations. - `training_set_ground_truth.json`: Gold standard for the 143-commit training set. - `training_set_first_classification.json`: First annotator's labels for the training IRR subset. - `training_set_second_classification.json`: Second annotator's labels for the training IRR subset. - `test_set_ground_truth.json`: Gold standard for the 384-commit test set. - `test_first_classification.json`: First annotator's labels for the training IRR subset. - `test_set_second_classification.json`: Second annotator's labels for the test IRR subset. - `tags_metadata.yaml`: Auxiliary metadata file used during preprocessing. - `README.md`: This file. - `requirements.txt`: Lists the required Python packages. How to Use This Package Setup 1. Create and activate a virtual environment (recommended). ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 2. Install required packages. ```bash pip install -r requirements.txt ``` Running the Analysis The Jupyter notebooks in the `code/` directory are numbered and named to be run in a logical sequence: Collection -> Preprocessing -> Analysis. We recommend following this order. - To reproduce our findings directly, you can start with the notebooks in `code/Analysis/`. They are configured to load the final, processed datasets provided in the `datasets/` folder. - To re-run the entire pipeline, start with the notebooks in `code/Collection/`, followed by `code/Preprocessing/`. Please note that running the full data collection and classification pipeline is time-consuming and may require significant computational resources and appropriate API keys for the LLM. Key Datasets Provided - For RQ1 & RQ2: `datasets/commits_datasets/HFCommitsClassification_final.csv` (100,000 models for RQ1; filtered to 14,343 models for RQ2). - For RQ3.1-3.3:`datasets/releases_datasets/HFReleasesClassification.csv` (2,251 releases from 202 models). - For RQ3.4: `datasets/releases_datasets/model_metadata.csv` (from 28 models). Contact If you have any questions or encounter issues with this package, please contact the corresponding author. If you find our work useful, please consider citing our paper.## 复现包：机器学习模型如何演化？ ### 概述 本复现包配套于论文《机器学习模型如何演化？》。本研究针对Hugging Face (HF)平台上的10万个模型的超68万次提交、以及其中202个模型的2251次版本发布，开展了大规模分析。本研究旨在通过基于精细化机器学习变更分类体系对提交类型进行分类，并利用贝叶斯网络分析其活动的时间模式，以探究机器学习（ML）模型的演化规律。 本研究主要围绕三个核心方向展开： 1. **提交变更分类**：对HF平台上的超96万次提交进行分类，详细拆解了变更类型及其分布情况。 2. **提交序列分析**：利用贝叶斯网络研究提交类型的序列与依赖关系，以识别时间演化模式。 3. **版本发布分析**：调研了版本发布类型的分布与演化过程，分析了模型属性与元数据在连续版本发布间的变化情况。 本复现包提供了复现论文中所有结果所需的全部代码、数据与文档资料。 ### 数据收集与预处理 #### 数据收集 我们通过Hugging Face Hub API从HF平台采集数据，数据采集截止至2025年5月，覆盖了当时平台上可用的超100万个模型。 - **模型与版本发布信息**：采集了抽样模型的元数据、提交历史以及版本发布信息（Git标签）。 - **详细提交变更**：为获取每次提交中修改文件的详细清单，我们采用了直接Git处理方案：对每个模型的代码仓库进行临时克隆，以编程方式提取每次提交SHA对应的变更文件列表。 #### 数据预处理 ##### 提交差异分析 我们计算了两次提交间关键JSON配置文件（如`config.json`）的差异，以识别新增、删除与更新的键值对，将其作为分类任务的输入特征。 ##### 提交分类 我们采用Bhatia等人提出的机器学习变更分类体系，结合Gemini 2.5 Flash大语言模型（LLM）对每一次提交进行分类。为确保该流程的可靠性，我们实施了严格的两阶段验证： 1. **提示词优化（训练阶段）**：基于143次提交的精选训练集，通过6轮迭代优化提示词。该流程以两名人工标注者构建的金标准数据集为参照（子集的人工-人工评分者间一致性：κ = 0.7798）。最终优化后的提示词与训练集金标准相比，Cohen's Kappa系数达到0.9068。 2. **最终验证（测试阶段）**：将优化后的提示词应用于独立的、具有统计显著性的384次提交样本集。与测试集金标准相比，大语言模型的分类结果Cohen's Kappa系数为0.8568；而测试集金标准本身的人工-人工评分者间一致性κ = 0.8150。 此外，我们还采用微调后的DistilBERT模型，将提交归类至Swanson提出的类别中，具体细节参见论文。 ### 文件夹结构 本复现包的组织形式如下，为保证清晰性，代码、数据与元数据均已分离： - `code/`：包含本研究使用的全部Jupyter笔记本。 - `Collection/`：用于从HF平台提取数据的脚本。 - `HFExtraction.ipynb`：采集核心模型与提交信息。 - `HFReleasesExtraction.ipynb`：采集版本发布（标签）的专属信息。 - `Preprocessing/`：用于数据清洗、处理与分类的脚本。 - `HFCommitsPreprocessing.ipynb`：处理提交数据、计算差异并准备用于分类与分析的数据集。 - `HFReleasesPreprocessing.ipynb`：处理并分类版本发布数据。 - `SwansonsClassification.ipynb`：将提交归类为修正型、完善型与适配型类别。 - `Analysis/`：用于复现各研究问题分析结果的Jupyter笔记本。 - `HFFileChanges.ipynb`：包含文件变更模式的初步分析。 - `RQ1_Analysis.ipynb`：针对研究问题1的分析。 - `RQ2_Analysis.ipynb`：针对研究问题2的分析。 - `RQ3_Analysis.ipynb`：针对研究问题3的分析。 - `datasets/`：包含分析笔记本中使用的核心最终数据集。 - `commits_datasets/`：包含经分类的主提交数据集。 - `HFCommitsClassification_final.csv`：用于研究问题1与2的超96万次已分类提交的最终数据集。 - `releases_datasets/`：包含与版本发布相关的数据集。 - `HFReleasesClassification.csv`：用于研究问题3的2251次已分类版本发布的最终数据集。 - `model_metadata.csv`：从模型文件中提取的内部元数据，用于研究问题3.4。 *注：此外还提供了其他中间CSV文件，以便无需从头开始即可重新运行分析的特定环节。* - `metadata/`：包含验证过程中使用的配置文件与数据。 - `validation_data/`：存储金标准数据集的子文件夹。 - `Agreement TOSEM Commit Changes.xlsx`：包含分类与验证流程细节的Excel表格。 - `prompt_refinement.txt`：最终经验证的、用于大语言模型分类的提示词，以及其历次迭代版本。 - `training_set_ground_truth.json`：143次提交训练集的金标准数据集。 - `training_set_first_classification.json`：训练集评分者间一致性子集的第一名标注者标注结果。 - `training_set_second_classification.json`：训练集评分者间一致性子集的第二名标注者标注结果。 - `test_set_ground_truth.json`：384次提交测试集的金标准数据集。 - `test_first_classification.json`：测试集评分者间一致性子集的第一名标注者标注结果。 - `test_set_second_classification.json`：测试集评分者间一致性子集的第二名标注者标注结果。 - `tags_metadata.yaml`：预处理阶段使用的辅助元数据文件。 - `README.md`：本说明文件。 - `requirements.txt`：列出所需的Python依赖包。 ### 本包使用方法 #### 环境搭建 1. 创建并激活虚拟环境（推荐）： bash python -m venv venv source venv/bin/activate # Windows系统请使用：venvScriptsactivate 2. 安装所需依赖包： bash pip install -r requirements.txt #### 运行分析 `code/`目录下的Jupyter笔记本已按逻辑顺序编号并命名，建议按照`数据采集 -> 预处理 -> 分析`的顺序运行： - 若需直接复现我们的研究结果，可直接从`code/Analysis/`目录下的笔记本开始，这些笔记本已配置为加载`datasets/`文件夹中提供的最终处理后数据集。 - 若需重新运行完整流程，请从`code/Collection/`目录下的笔记本开始，随后依次运行`code/Preprocessing/`目录下的脚本。请注意，完整的数据采集与分类流程耗时较长，可能需要大量计算资源以及适配大语言模型所需的API密钥。 #### 核心数据集说明 - 针对研究问题1与2：`datasets/commits_datasets/HFCommitsClassification_final.csv`（研究问题1使用10万个模型；研究问题2经筛选后使用14343个模型）。 - 针对研究问题3.1~3.3：`datasets/releases_datasets/HFReleasesClassification.csv`（来自202个模型的2251次版本发布）。 - 针对研究问题3.4：`datasets/releases_datasets/model_metadata.csv`（来自28个模型）。 ### 联系方式 若您对此复现包有任何疑问或遇到问题，请联系通讯作者。若您认为本研究成果有价值，请考虑引用我们的论文。