Dataset of the paper: "The Transparency of Pre-trained Deep Learning Models: An Empirical Study on Hugging Face"

This replication package contains datasets and scripts related to the paper: "*The Transparency of Pre-trained Deep Learning Models: An Empirical Study on Hugging Face*" ## Root directory<br> - `statistics.r`: R script used to compute the RQ1 correlation and the RQ2/RQ3 inter-rater agreements<br> - `modelsInfo.zip`: zip file containing all the downloaded model cards (in JSON format)<br> - `script`: directory containing all the scripts used to collect and process data. For further details, see README file inside the script directory. ## RQ1<br> - `RQ1/RQ1_HF-models-list.csv`: list of HF models analyzed<br> - `RQ1/RQ1_github-prj-list.txt`: list of GitHub projects using the *transformers* library<br> - `RQ1/RQ1_github-Prj_model-Used.csv`: contains usage pairs: project, model<br> - `RQ1/RQ1_prj-num-models-reused.csv`: number of models used by each GitHub project<br> - `RQ1/RQ1_model-download_num-prj_correlation.csv` contains, for each model used by GitHub projects: the name, the task, the number of reusing projects, and the number of downloads <br> ## RQ2<br> - `RQ2/RQ2_dataset-list.txt`: list of HF datasets<br> - `RQ2/RQ2_datasetSample.csv`: sample set of models used for the manual analysis of datasets<br> - `RQ2/analyzeDatasetTags.py`: Python script to analyze model tags for the presence of datasets. it requires to unzip the `modelsInfo.zip` in a directory with the same name (`modelsInfo`) at the root of the replication package folder. Produces the output to stdout. To redirect in a file fo be analyzed by the `RQ2/countDataset.py` script<br> - `RQ2/countDataset.py`: given the output of `RQ2/analyzeDatasetTags.py` (passed as argument) produces, for each model, a list of Booleans indicating whether (i) the model only declares HF datasets, (ii) the model only declares external datasets, (iii) the model declares both, and (iv) the model is part of the sample for the manual analysis<br> - `RQ2_datasetTags.csv`: output of `RQ2/analyzeDatasetTags.py`<br> - `RQ2_dataset_usage_count.csv`: output of `RQ2/countDataset.py` <br> ## RQ3<br> - `RQ3/tableBias.pdf`: table detailing the number of occurrences of different types of bias by model Task<br> - `RQ3/RQ3_bias_classification_sheet.csv`: results of the manual labeling<br> - `RQ3/RQ3_isBiased.csv`: file to compute the inter-rater agreement of whether or not a model documents Bias<br> - `RQ3/RQ3_biasAgrLabels.csv`: file to compute the inter-rater agreement related to bias categories<br> - `RQ3/RQ3_final_bias_categories_with_levels.csv`: for each model in the sample, this file lists (i) the bias leaf category, (ii) the first-level category, and (iii) the intermediate category <br> ## RQ4<br> - `RQ4/RQ4_{NETWORK-RESTRICTIVE|RESTRICTIVE|WEAK-RESTRICTIVE|PERMISSIVE}-license-list.txt`: lists of licenses with different permissiveness<br> - `RQ4/RQ4_prjs_license.csv`: for each project linked to models, among other fields it indicates the license tag and name<br> - `RQ4/RQ4_models_license.csv`: for each model, indicates among other pieces of info, whether the model has a license, and if yes what kind of license<br> - `RQ4/RQ4_model-prj-license_contingency_table.csv`: usage contingency table between projects' licenses (columns) and models' licenses (rows)<br> - `RQ4/RQ4_models_prjs_licenses_with_type.csv`: pairs project-model, with their respective licenses and permissiveness level ## scripts<br> Contains the scripts used to mine Hugging Face and GitHub. Details are in the enclosed README