Replication package for the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models"

This replication package contains datasets and scripts related to the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models" root directory statistics.r: R script used to compute the correlation between usage and downloads, and the RQ1/RQ2 inter-rater agreements scripts/ Directory containing all the scripts used to collect and process data. For further details, see the README file inside the script directory. further-analyses/ Directory containing scripts and data used for further analyses. For further details, see the README file inside the directory. first-snaphot Contains data related to the first snapshot (ICPC 2024 study) modelsInfo.zip: zip file containing all the downloaded model cards (in JSON format) Dataset Dataset/DatasetHFmodellist.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipelinetag,pipelinecategory,License,licensemodelpermissivity Dataset/DatasetGitHubprjlistTransformers.txt: list of GitHub projects using the transformers library Dataset/DatasetGitHubprjlistDiffusers.txt: list of GitHub projects using the diffuserslibrary Dataset/DatasetGitHubprjfrompretrainedTransformers.txt: list of GitHub projects using the "frompretrained" transformers library Dataset/DatasetGitHubprjfrompretrainedDiffusers.txt: list of GitHub projects using the "frompretrained" diffusers library Dataset/DatasetGitHubprjmodelusedTransformers.csv: contains usage pairs: project, model for transformers library Dataset/DatasetGitHubprjmodelusedDiffusers.csv: contains usage pairs: project, model for diffusers library Dataset/DatasetIntersectedModels.csv : contains the models shared between the first and second snapshot for category Dataset/modelsReadme: contains the model cards belonging to the sample size Dataset/projectswith5ormorestars.csv: contains the projects with numStars major of 5 Dataset/projectsstars_summary.csv: contains the number of total projects with numStars RQ1 RQ1/RQ1datasetlistHF.txt: list of HF datasets RQ1/RQ1datasetTags.txt: list of models declaring the dataset tag RQ1/RQ1modelDataset.csv : list of models declaring the dataset tag with their respective datasets RQ1/RQ1datasetSample.csv: sample set of models used for the manual analysis of datasets RQ2 RQ2/RQ2biasclassification_sheet.csv: results of the manual labeling RQ3 RQ3/RQ3LicenseModels.csv: model license list,categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3prjmodellicensepermissivityTransformersDiffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffuserslibrary RQ3/RQ3prjmodellicensepermissivityTransformersDiffusersStarmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5 RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffuserslibrary in terms of licenses RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusersStarmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5 RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5 RQ4 RQ4/RQ4TermsofUseSnapshot1.csv: results of the manual labeling related to terms of use for the first snapshot RQ4/RQ4TermsofUseSnapshot2.csv: results of the manual labeling related to terms of use for the second snapshot Dataset Dataset/Dataset_HF-models-list.csv: list of HF models analyzed Dataset/Dataset_github-prj-list.txt: list of GitHub projects using the transformers library Dataset/Dataset_github-Prj_model-Used.csv: contains usage pairs: project, model Dataset/Dataset_prj-num-models-reused.csv: number of models used by each GitHub project Dataset/Dataset_model-download_num-prj_correlation.csv contains, for each model used by GitHub projects: the name, the task, the number of reusing projects, and the number of downloads RQ1 RQ1/RQ1_dataset-list.txt: list of HF datasets RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets RQ1/RQ1_analyzeDatasetTags.py: Python script to analyze model tags for the presence of datasets. it requires to unzip the modelsInfo.zip in a directory with the same name (modelsInfo) at the root of the replication package folder. Produces the output to stdout. To redirect in a file fo be analyzed by the RQ2/countDataset.py script RQ1/RQ1_countDataset.py: given the output of RQ2/analyzeDatasetTags.py (passed as argument) produces, for each model, a list of Booleans indicating whether (i) the model only declares HF datasets, (ii) the model only declares external datasets, (iii) the model declares both, and (iv) the model is part of the sample for the manual analysis RQ1/RQ1_datasetTags.csv: output of RQ2/analyzeDatasetTags.py RQ1/RQ1_dataset_usage_count.csv: output of RQ2/countDataset.py RQ2 RQ2/tableBias.pdf: table detailing the number of occurrences of different types of bias by model Task RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling RQ2/RQ2_isBiased.csv: file to compute the inter-rater agreement of whether or not a model documents Bias RQ2/RQ2_biasAgrLabels.csv: file to compute the inter-rater agreement related to bias categories RQ2/RQ2_final_bias_categories_with_levels.csv: for each model in the sample, this file lists (i) the bias leaf category, (ii) the first-level category, and (iii) the intermediate category RQ3 RQ3/RQ3_LicenseValidation.csv: manual validation of a sample of licenses RQ3/RQ3_{NETWORK-RESTRICTIVE|RESTRICTIVE|WEAK-RESTRICTIVE|PERMISSIVE}-license-list.txt: lists of licenses with different permissiveness RQ3/RQ3_prjs_license.csv: for each project linked to models, among other fields it indicates the license tag and name RQ3/RQ3_models_license.csv: for each model, indicates among other pieces of info, whether the model has a license, and if yes what kind of license RQ3/RQ3_model-prj-license_contingency_table.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) RQ3/RQ3_models_prjs_licenses_with_type.csv: pairs project-model, with their respective licenses and permissiveness level RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types: analysis\_by\_type.py: script to perform the analysis list.csv: the list of licenses with their permissiveness licensebytype.csv: licenses of all analyzed models RQ4 RQ4/RQ4_Terms_of_Use_Snapshot1.csv: results of the manual labeling related to terms of use for the first snapshot second-snaphot Contains data related to the second snapshot. Dataset Dataset/Dataset_HF_model_list.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipeline_tag,pipeline_category,License,license_model_permissivity Dataset/Dataset_GitHub_prj_list_Transformers.txt: list of GitHub projects using the transformers library Dataset/Dataset_GitHub_prj_list_Diffusers.txt: list of GitHub projects using the diffusers library Dataset/Dataset_GitHub_prj_frompretrained_Transformers.txt: list of GitHub projects using the "from_pretrained" transformers library Dataset/Dataset_GitHub_prj_frompretrained_Diffusers.txt: list of GitHub projects using the "from_pretrained" diffusers library Dataset/Dataset_GitHub_prj_model_used_Transformers.csv: contains usage pairs: project, model for transformerslibrary Dataset/Dataset_GitHub_prj_model_used_Diffusers.csv: contains usage pairs: project, model for diffusers library Dataset/Dataset_IntersectedModels.csv : contains the models shared between the first and second snapshot for category Dataset/modelsReadme: contains the model cards belonging to the sample size Dataset/projects_with_5_or_more_stars.csv: contains the projects with numStars major of 5 Dataset/projects_stars_summary.csv: contains the number of total projects with numStars RQ1 RQ1/RQ1_dataset_list_HF.txt: list of HF datasets RQ1/RQ1_datasetTags.txt: list of models declaring the dataset tag RQ1/RQ1_modelDataset.csv : list of models declaring the dataset tag with their respective datasets RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets RQ2 RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling RQ3 RQ3/RQ3_License_Models.csv: model license list,categorized by permissiveness, with the respective number of occurrences RQ3/RQ3_License_prjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3_License_prjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers_Starmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5 RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5 RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5 RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types: analysis\_by\_type.py: script to perform the analysis list.csv: the list of licenses with their permissiveness licensebytype.csv: licenses of all analyzed models RQ4 RQ4/RQ4_Terms_of_Use_Snapshot2.csv: results of the manual labeling related to terms of use for the second snapshot