Replication package of the paper "Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software"

Replication Package This replication package contains the necessary data and Python script for reproducing the results of our paper: "Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software". The goal of the study is to analyze automated decision-making processes in open-source, high-risk ML projects. It involves gathering data from GitHub, comparing it to the dependents of huggingface/transformers, manually reviewing the project's workflow, and clustering the post-processing and outputs of various projects. Project Structure The project is organized into the following main directories and files: data/: Contains datasets used in the project dependents_transformers.csv: This file contains a pre-sorted list of dependents of the huggingface/transformer library on GitHub. For each client project the file reports: (i) name of the project, (ii) number of stars, and (iii) number of forks. github_domain_repos.csv: This file contains the set of projects we have retrieved by searching by topics (application areas) using the GitHub API. For each project we report: repoName: Name of the repository (owner name/project name). stars: Number of stars. forks: Number of forks. total_commits: Number of total commits. lastcommitmonth: Month of last commit. lastcommitauthor: Author of last commit. lastcommitmessage: Commit message of the last commit. language: The primary programming language used in the repository. github_link: The URL to access the repository on GitHub. domain: The specific domain/topic under which the repository is categorized. sub-field: The specific sub-domain under which the domain is sub-categorized. overlap_github_transformers.csv: This file is the result of the intersection between the high risk projects (github_domain_repos.csv) and the client projects importing the huggingface/transformer library (sorted_dependents_repo_huggingface_transformers.csv). It lists the initial sample accounting for 173 projects used to answer our research questions. python/: Contains Python scripts and dependencies. requirements.txt: Lists the Python package dependencies required for the project. .env: Holds environment variables, including the GitHub token. example.env: Example environment file to copy and populate with your GitHub token. domain_wise_repository_github.py: Python script queries GitHub by domain/topic and saves repository metadata. overlapGithubWithTransformersDep.py: Python script compares domain-fetched repositories with the HuggingFace dependents CSV and produces matched output. results/: Contains output files from the analysis. github_systems_domain_wise_workflow.pdf: Extended workflow of a GitHub repository from input to output: each step includes input, pre-processing, decision, post-processing, and output. automated_decision_types.pdf: Contains clustering outputs of GitHub projects and their explanations. postprocessing_rulebase_cluster.pdf: Describes the rule-based post-processing applied to clusters and selection logic. final_decision_making_repos.csv: This file contains the final list of selected GitHub repositories. Each row represents one project used in the decision-making analysis. domain: Main application area of the project (such as healthcare or finance). repoName: Short name of the GitHub repository. dependent: Shows whether the project depends on other tools, models, or systems. sub-field: Specific area within the main domain. repository_name: Full or official name of the repository. domain_name: standardized name of the project domain. subdomainname: standardized name of the project sub-domain. decision_type: Type of decision the system performs TraditionalMLModel/ Logical: Indicates use of traditional machine learning or rule-based logic. Transformer/DeepLearningModel: Shows whether deep learning or transformer models are used. LLM_Used: Name of the large language model used in the project, if any. modeloutputusage: How the model’s output is used to make final decisions. post-processing_type: Type of rules or logic applied after the model output. cluster : Cluster group assigned to the project based on its functionality. automation_level : Level of automation (manual, semi-automatic, or fully automatic). termofuse_restricted(yes/no) : Indicates whether the repository has usage or violation of term of use. models : List of AI or ML models used in the project. appendix/: Contains additional analysis documents. github_projects_domain_sub-domains_analysis.pdf: Contains a table that illustrates the relationship between the main domains and their corresponding sub-domains. output_categories.pdf: This table summarizes the output-based categorization used in our study to classify GitHub projects that employ machine learning models for automated or semi-automated decision-making. Category Name: the label used in the study to identify a class of decision outputs; Keywords: representative terms observed in documentation or code that signal the presence of this output type; Description: a concise explanation of the kind of decision or artifact generated by projects in this category. unique_models_github_systems_analysis.pdf: This table lists the unique machine learning models identified across the analyzed GitHub repositories and reports how often each model appears as a dependency. For each model, we show the repositories in which it is used and the corresponding usage count. Setup Instructions Ensure you have Python 3.8+ installed on your Windows machine. Obtain a GitHub personal access token for higher API rate limits. Install the required dependencies by running: pip install -r python/requirements.txt Set up your environment by copying the example environment file and editing it to include your GitHub token:cd python copy example.env .env # Edit .env and set GITHUB_TOKEN=ghp_... Ensure the required data files exist in the data folder. How to Run Fetch domain repositories: cd python python domain_wise_repository_github.py Compare fetched domain repos with the pre-sorted dependents CSV:python compare_sorted_dependents_repo_huggingface_with_git_domain_wise_data.py复现包 本复现包包含用于复现我们论文《面向野外场景的机器学习：开源软件中不合规机器学习自动化的早期证据》研究结果的必要数据与Python脚本。本研究旨在分析开源高风险机器学习项目中的自动化决策流程，具体工作包括从GitHub平台采集数据，将其与huggingface/transformers库的依赖项目进行比对，手动审查项目工作流，并对各类项目的后处理流程与输出结果进行聚类分析。 ## 项目结构 本项目按以下主要目录与文件组织： ### data/：存放项目使用的数据集 1. `dependents_transformers.csv`：该文件包含GitHub平台上huggingface/transformers库依赖项目的预排序列表。针对每个客户端项目，该文件记录了：(i) 项目名称；(ii) Star数量；(iii) Fork数量。 2. `github_domain_repos.csv`：该文件包含我们通过GitHub API按主题（应用领域）检索得到的项目集合。针对每个项目，我们记录了以下信息： - repoName：仓库名称（所有者名称/项目名称） - stars：Star数量 - forks：Fork数量 - total_commits：总提交次数 - lastcommitmonth：最近一次提交的月份 - lastcommitauthor：最近一次提交的作者 - lastcommitmessage：最近一次提交的提交信息 - language：仓库使用的主要编程语言 - github_link：该仓库在GitHub平台的访问链接 - domain：该仓库所属的具体领域/主题 - sub-field：该领域下的具体子领域 3. `overlap_github_transformers.csv`：该文件为高风险项目（来自`github_domain_repos.csv`）与导入huggingface/transformers库的客户端项目（来自`sorted_dependents_repo_huggingface_transformers.csv`）的交集结果，列出了用于回答研究问题的173个初始样本项目。 ### python/：存放Python脚本与依赖项 1. `requirements.txt`：列出项目所需的Python包依赖 2. `.env`：存储环境变量，包括GitHub访问令牌 3. `example.env`：示例环境文件，用于复制并填入个人GitHub访问令牌 4. `domain_wise_repository_github.py`：用于按领域/主题查询GitHub并保存仓库元数据的Python脚本 5. `overlapGithubWithTransformersDep.py`：用于将按领域检索得到的仓库与HuggingFace依赖项目CSV文件进行比对并生成匹配结果的Python脚本 ### results/：存放分析输出文件 1. `github_systems_domain_wise_workflow.pdf`：GitHub仓库从输入到输出的扩展工作流文档，每个步骤均包含输入、预处理、决策、后处理与输出环节 2. `automated_decision_types.pdf`：包含GitHub项目的聚类结果及其说明文档 3. `postprocessing_rulebase_cluster.pdf`：描述针对聚类结果应用的基于规则的后处理流程与选择逻辑 4. `final_decision_making_repos.csv`：包含最终选定的GitHub仓库列表，每行代表一个用于决策分析的项目，各字段说明如下： - domain：项目的主要应用领域（如医疗、金融等） - repoName：GitHub仓库的简称 - dependent：标识项目是否依赖其他工具、模型或系统 - sub-field：主领域下的具体细分领域 - repository_name：仓库的完整/官方名称 - domain_name：项目领域的标准化名称 - subdomainname：项目子领域的标准化名称 - decision_type：系统执行的决策类型 - TraditionalMLModel/Logical：标识是否使用传统机器学习或基于规则的逻辑 - Transformer/DeepLearningModel：标识是否使用深度学习或Transformer模型 - LLM_Used：项目中使用的大语言模型（Large Language Model）名称（如适用） - modeloutputusage：模型输出如何用于生成最终决策 - post-processing_type：模型输出后应用的规则或逻辑类型 - cluster：基于项目功能分配的聚类组 - automation_level：自动化等级（手动、半自动或全自动） - termofuse_restricted(yes/no)：标识该仓库是否存在使用限制或违反使用条款的情况 - models：项目中使用的人工智能或机器学习模型列表 ### appendix/：包含额外的分析文档 1. `github_projects_domain_sub-domains_analysis.pdf`：包含展示主领域与对应子领域之间关系的表格 2. `output_categories.pdf`：该表格汇总了本研究中用于对采用机器学习模型实现自动化或半自动化决策的GitHub项目进行分类的基于输出的分类体系，各分类项说明如下： - Category Name：研究中用于标识一类决策输出的标签 - Keywords：在文档或代码中观察到的、用于指示该输出类型存在的代表性术语 - Description：对该类别项目生成的决策或产物的简要说明 3. `unique_models_github_systems_analysis.pdf`：该表格列出了在所分析的GitHub仓库中识别到的唯一机器学习模型，并报告了每个模型作为依赖项的出现频率，同时展示了使用该模型的仓库及其对应使用次数。 ## 安装说明 请确保你的Windows设备上已安装Python 3.8及以上版本。 获取GitHub个人访问令牌以提升API调用限额。 通过执行以下命令安装所需依赖：`pip install -r python/requirements.txt` 通过复制示例环境文件并编辑以填入GitHub访问令牌来配置环境： bash cd python copy example.env .env # 编辑.env文件，设置GITHUB_TOKEN=ghp_... 请确保`data`文件夹中存在所需的数据文件。 ## 运行方法 1. 获取按领域划分的仓库：`cd python && python domain_wise_repository_github.py` 2. 将获取的领域仓库与预排序的依赖项目CSV文件进行比对：`python compare_sorted_dependents_repo_huggingface_with_git_domain_wise_data.py`