Replication Package for "Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings"

Overview This replication package supports the paper "Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings". The goal of the study is to analyze Hugging Face (HF) Pre Trained Models (PTMs) for the purpose of their automatic classification with respect to their application to Software Engineering (SE) activities and tasks from the point of view of software engineers in the context of model selection. Our repository mining study followed a five-phase pipeline: (i) identification SE tasks from the literature; (ii) collection of PTM data from the HF API, including model card descriptions and metadata, and the abstracts of the associated arXiv papers; (iii) text processing to ensure consistency; (iv) a two-phase validation of SE relevance, involving humans and LLM assistance, supported by five pilot studies with human annotators and a generalization test; (v) and data analysis. This process yielded a validated catalogue of 2,205 SE PTMs. Repository Structure It repository contains a README file and scripts with detailed instructions for replicating the study. The package also contains all relevant data, including the raw dataset, human validation annotations, and the final curated dataset. /analysis ├── [Jupyter notebooks for analyzing PTMs, including the figures presented in the paper] /code └── [Pythons scripts and Jupyter notebooks for collecting, preparing and filtering the data] ├── llm/ │ └── [Pilot tests using Gemini 2.0 FLash followed by full-scale evaluation on entire dataset] /data ├── data_collection/ │ └── [Raw data collected from Hugging Face] ├── data_filtering/ │ └── [Filtered data based on selection criteria] ├── data_search/ │ └── [Data obtained through search processes] ├── related_work │ └── [Data from HFCommunity and PeaTMOSS] ├── subset_LLM_generalization/ | └── [Data subsets for testing the generalizability of the LLM's prompt] └── subset_pilot/ └── [Data subsets for pilot experiments, which were annotated by 3 independent researchers] Setup To replicate the analysis or run the pipeline, clone this repository and install the required dependencies: pip install -r requirements.txt Note: Access to the Hugging Face API requires an API key. The pipeline uses Gemini 2.0 Flash, which requires an API key and may incur costs. Pipeline Description 1) Task Identification - code/define_SE_tasks.ipynb Defines the SE activities and tasks used in this study. The resulting list is saved as code/se_tasks_lemmatized.csv 2) Data Collection & 3) Data Processing - code/get_mdls_tasks.py Uses the Hugging Face API to retrieve all available PTMs and extract any declared tasks. Results saved in data/data_collection/mdls_se_tasks.csv - code/get_mdls_tasks_abstract.py Similarly extracts task information from the abstracts of associated arXiv papers. Results saved in data/data_collection/mdls_se_tasks_abstract.csv 3.2) Task Search - analysis/data_search.ipynb Selects PTMs mentioning any SE task based on both data sources. Results saved in data/data_search/mdls_with_tasks.csv 3.3) Outlier Detection - code/filtering_LOG-DEB-COD.py Detects outliers in the previous step. Results saved in data/data_filtering/mdls_with_tasks-LOG-DEB-COD-filtered.csv - analysis/data_filtering.ipynb Filters outliers and enriches data with metadata such as SE activity and PTM creation date. Results saved in data/data_filtering/mdls_with_tasks-LOG-DEB-COD-filtered_activities.csv 3.4) Unique Card Detection - code/filtering_duplicate_cards.py Detects entries with highly similar model cards for a second filtering step. Results saved in data/data_filtering/mdls_with_tasks-LOG-DEB-COD-filtered_activities_no-duplicate-cards-99-trace.csv - analysis/data_filtering_duplicates.ipynb Analyzes filtered data. 4) LLM-Based Validation 4.1) Prompts Refinement Selected PTMs are classified into specific SE activities using Gemini 2.0 Flash LLM: Requirements Engineering (RE), Software Design (SD), Software Implementation (SI), Software Quality Assurance (SQA), and Software Maintenance (SM). Each activity has a dedicated folder inside code/llm containing notebooks for pilot tests and final runs: code/llm/{se_activity}/{se_activity}_gemini_pilot_all.ipynb, with data saved in {se_activity}_gemini_all.csv The agreement between annotators can be found in code/llm/manual_agreement.ipynb - code/get_mdls_HF_attributes.ipynb Enriches the LLM-confirmed SE PTMs with Hugging Face metadata (tags, downloads, likes, etc.). Data saved under code/llm/{se_activity}/{se_activity}_gemini_all_HFinsights.csv - code/merge_mapping_HF_attributes.ipynb Combines LLM classifications, HF metadata, and activity mappings into CSV files per SE activity. Results saved as code/llm/{se_activity}/{se_activity}_gemini_all_HFinsights_merged.csv - code/get_HF_attributes_renamed_mdls.ipynb Detects PTMs that have been renamed, updates their name and gets HF attributes for them. Results saved as code/llm/{se_activity}/{se_activity}_gemini_all_HFinsights_merged.csv - code/llm/combine_llm_results.ipynb Merges all per-activity CSV files into a final consolidated dataset: data/data_filtering/mdls_with_tasks-LOG-DEB-COD-filtered_activities_no-duplicate-cards-99-trace_LLM.csv 4.2) LLM Generalization Test - data/subset_LLM_generalization/generalization_subset%20-%20subset.csv A manually annotated subset is used to verify that the LLM prompt generalizes well and does not overfit. This is analyzed in code/llm/manual_agreement_generalization.ipynb