This is the replication package of the manuscript "An Empirical Study on the Integration of Scientific Contributions into Machine Learning Frameworks"

This replication contains the datasets and scripts associated with the paper: "An Empirical Study on the Integration of Scientific Contributions into Machine Learning Frameworks" Directory Structure analysis_matched_keywords This directory contains the manual analysis performed on all Pull Requests (PRs) from the six analyzed machine learning frameworks that matched the keywords identified in the study. (Diffusers_Analysis_matched.csv, Keras_Analysis_matched.csv, Pytorch_Analysis_matched.csv, SkLearn_Analysis_matched.csv, Tensorflow_Analysis_matched.csv, Transformers_Analysis_matched.csv) For each framework, the provided CSV files include the following information: Number_PR: Unique identifier/number of the Pull Request. URL_PR: Direct link to the PR on GitHub. Scientific_Paper(Y/N): A binary flag (Y for Yes, N for No) indicating whether the PR body contain an explicit link or formal reference to a scientific paper or academic article. Note: Contains the direct links (URLs) to the scientific contributions, papers, or articles identified during the manual review. Replication(Y/N): Evaluated only if Scientific_Paper is Y. It indicates whether the referenced paper explicitly cites or provides a replication package (e.g., a link to a code repository, dataset, or supplemental materials). LinkRepo: Populated only when Replication (Y/N) is Y. It contains the direct URL to the identified replication package or the official source code repository associated with the scientific paper. Contributor(Y/N): Evaluated only when LinkRepo is present. Indicates whether the replication package was used to directly contribute new functionalities (algorithms, modules, enhancements) to the ML framework (Y), as opposed to merely using the framework to reproduce paper results (N). Language: Populated only when Contributor (Y/N) is Y. It identifies the primary programming language used in the associated replication package (e.g., Python, C++, etc.). State: Populated only if Contributor is Y. The final status of the Pull Request (Merged, Closed, or Open). Title: Original GitHub metadata (Title) Description: Original GitHub metadata (Body text) User: GitHub handle of the PR author. Created_At: Timestamps tracking the PR lifecycle. Closed_At: Timestamps tracking the PR lifecycle. Merged_At: Populated only if Contributor is Y. The timestamp indicating when the contribution was officially integrated into the framework. Comments: Contains the direct URLs to the discussions and comments associated with the Pull Request. Keyword_Matched: The specific keyword that triggered the initial inclusion of this PR. The directory also contains: KEYWORDS.txt: A plain text file containing the complete list of keywords used for the initial search across the six machine learning repositories. Final_PRs.csv : It contains the final 63 Pull Requests identified after applying all filtering layers (Scientific Reference, Replication Package availability, Python language, and Merged status). These PRs represent the direct integration of scientific research into the analyzed ML frameworks. results This directory contains the output of our analysis, organized into subdirectories corresponding to each Research Question (RQ) addressed in the study. results/RQ1 Contains: RQ1_categories.csv: Contains the detailed manual analysis performed by two independent reviewers. It includes the individual classifications and the final categorization reached after a conflict resolution phase. This file identifies the specific type of change each PR introduced RQ1_macro.csv: Provides an abstraction of the data into broader macro-categories. It includes the total occurrences and frequency of each category across the dataset. results/RQ2 Contains: RQ2_role.csv: Contains the detailed profile classification of the users associated with the 63 selected Pull Requests. results/RQ3 Contains: RQ3_time_lag.csv: This file tracks the temporal evolution of the 63 selected Pull Requests. All delta-time columns are calculated in days. scientific_vs_other directory: This directory contains the datasets used to perform the statistical comparison between scientific and non-scientific contributions. These files served as the input for the Wilcoxon Rank-Sum Test. The directory is organized by framework, with two files for each (e.g., diffusers_scientific.csv and diffusers_other.csv). For each file, the key metric used for the analysis is: Time (Merged-Created): This column represents the duration (in days) from the opening of the PR to its successful merge. Methodological Note: We extracted the Time (Merged-Created) values from both the scientific and not_scientific groups for each framework to conduct the Rank-Sum Test results/RQ4 Contains: total_comments.csv : A comprehensive dataset of all comments extracted from the analyzed PRs, categorized by type: Close, Merge, PC (Pull Comment), RC (Review Comment), DESCR (Description), and Commit. comments_PC_RC.csv : A filtered subset containing only Pull Comments (PC) and Review Comments (RC), used as the input for the Ollama classification. comments_PC_RC_ollama_category.csv : The output of the LLM classification, matching each PC/RC comment with its respective category assigned by Ollama. PR_Sample_Shuffled.csv: A randomized sample of comments used for the manual validation phase. It includes the independent classifications from two reviewers and the final consensus reached after conflict resolution. prompt_ollama.txt : The specific system prompt and instructions provided to the Ollama model to guide the classification process. results/RQ5 Contains: diff_analysis directory: Contains individual CSV files for each framework. Each file stores the diff (code changes) for every identified Pull Request. tmp_nicad_2_[framework] : Temporary work directories containing the raw output and intermediate files generated by the NiCad clone detection tool for each respective framework. RQ5_Nicad.csv: A summary report of the cross-clones detected by Nicad. RQ5_copyright.csv: A detailed list of Pull Requests where explicit copyright notices were identified within the code diffs. scripts This directory contains the Python an R scripts used for data collection, processing, and statistical analysis. Data Collection & Extraction matched_keywords.py: Scans the PR bodies across the six frameworks to identify entries containing specific keywords (e.g., arxiv, paper, doi). download_pr_closed.py: Interacts with the GitHub API to download the full history of closed Pull Requests for a given repository. diff.py: Retrieves the specific code diffs for each identified PR via GitHub API. download_PR_comments.py: Extracts all PR-related interactions, including Pull Comments (PC), Review Comments (RC), descriptions, commit history, and merge/close events. Processing & Analysis Ollama.py: Automates the categorization of comments (PC/RC) using the Ollama LLM. copyright.py: Scans code diffs for new or modified lines containing the keyword "copyright" to identify legal/origin markers. Nicad.py: Automates the clone detection process using the NiCad tool. parser_crossrepo_nicad.py: Filters the NiCad output to isolate cross-repository clones. Statistical Analysis rank_sum_test.R: Performs the statistical comparison between "scientific" and "other" PRs using: Wilcoxon Rank-Sum Test: To assess significant differences in merge time distributions. Cliff's Delta: To quantify the magnitude (effect size) of the differences observed.