An Empirical Study on the Accuracy of GitHub's Dependency Graph and the Nature of Its Inaccuracy: A Replication Package

Replication Package This is the Replication Package for the paper:“An Empirical Study on the Accuracy of GitHub's Dependency Graph and the Nature of Its Inaccuracy” Below is a detailed explanation of the directory structure, contents, and instructions for running the scripts to replicate the study. Prerequisites Python ≥ 3.8 R (for statistical analysis) GitHub Personal Access Token (used in API-based scripts) Required Python packages: pandas, openpyxl, requests, etc. Before starting a new analysis, delete all previously generated files in the assets directories. This avoids issues like “file already exists” or name conflicts during script execution. Directory Structure ▸ java/ & python/ Each of these contains data and scripts specific to Java and Python repositories, respectively. java/ ├── assets/ # Input/output CSV/XLSX files └── script/ # Scripts to gather and analyze data python/ ├── assets/ └── script/ Subfolders: assets/: Contains datasets and analysis results. script/: Python scripts to extract, filter, analyze, and validate GitHub dependency data. ▸ rq1_analysis/ Contains scripts and output files used for analyzing Research Question 1 (RQ1). rq1_analysis/ ├── scripts/ │ ├── rq1_analysis.py │ └── statistics.R └── assets/ ├── java_repo_metrics.xlsx └── python_repo_metrics.xlsx ▸ rq2_analysis/ Contains scripts, results, and samples related to Research Question 2 (RQ2). rq2_analysis/ ├── assets/ │ ├── *.csv │ └── *.xlsx ├── sample_data/ │ ├── *.csv │ └── *.xlsx └── script/ ├── 1.bw_sample_extraction.py └── ... Script Execution Order Each language has its own scripts but they follow the same logic: Generate a GitHub token and add it to scripts (InsertYourToken) Run scripts in numerical order (e.g., 1_, 2_, … 11_) Inspect output files in /assets after each stage Scripts handle dependency graph retrieval, forward/backward analysis, and manifest/lock file matching. Data Gathering and Analysis Java and Python Data Gathering Folder: /{prg_lang}/ Scripts: 1_dependencies_repositories_retriever.py : Retrieves dependency's repositories from GitHub Dependency Graph. 2_backward_analysis.py : Performs backward analysis on the retrieved data. 3_dependencies_filtering.py : Filters dependencies based on specific criteria (e.g., Package Manager). 4_backward_results_analysis.py: Extract a summary of the backward analysis aggregate by error cases. 5_dependents_repositories_retriever.py: Retrieves dependent's repositories from GitHub Dependency Graph. 6_dependents_count.py: Counts the number of dependents for each repository. 7_forward_analysis.py: Performs forward analysis on the retrieved data. 8_forward_results_analysis.py: Extract a summary of the forward analysis aggregate by error cases. 9_manifest_lock_extraction.py: Extracts the manifest/lock file information from the dependencies retrieved in the 1_dependencies_repositories_retriever.py by GitHub's dependency graph. 10_manifest_lock_dependencies_extraction.py: Retrieves raw file of the manifest/lock file from GitHub and then parse these to extract the dependencies. 11_manifest_lock_to_dg_match.py: Matches the dependencies retrieved from manifest/lock files with the dependencies retrieved from GitHub's dependency graph. RQ1 Analysis Folder: /rq1_analysis/ Scripts: rq1_analysis.py: Executes RQ1 analysis pipeline statistics.R: Performs Chi-squared, Fisher’s Exact Test, Krippendorff’s alpha Output: Excel files (e.g., *_repo_metrics.xlsx, *_match_count.xlsx) with per-language statistics used in the paper’s tables and figures. RQ2 Analysis Folder: /rq2_analysis/ This folder contains data and scripts used to validate and analyze the findings for Research Question 2, especially around error types and their frequency. Structure: rq2_analysis/ ├── assets/ # Intermediate and final results (CSV/XLSX) │ ├── bw_occurence_not_found.csv │ ├── fw_analysis_results.csv │ ├── kripp_test_bw.xlsx │ └── ... │ ├── sample_data/ # Data samples for manual verification and statistics │ ├── bw_analysis.csv │ ├── non_match_sample.csv │ └── ... │ └── script/ # Python scripts used for sampling, result aggregation, and enrichment ├── 1.bw_sample_extraction.py ├── 5.fw_sample_extraction.py ├── 8.fw_final_results.py └── 10.match_add_link.py Scripts: 1.bw_sample_extraction.py: Random sampling of backward error cases 2.bw_sample_dep_link.py: Add dependency links for backward analysis 3.bw_pkg_not_found_analysis.py: Analyze packages not found in backward analysis 4bw_pkg_not_found_results.py: Generate results for packages not found 5.fw_sample_extraction.py: Random sampling of forward error cases 6.fw_analysis.py: Processes forward analysis output 7.fw_res_manipulation.py: Manipulates forward analysis results 8.fw_final_results.py: Finalizes forward analysis results 9.match_add_date.py: Adds temporal dimension to matching results 10.match_add_date.py: Adds date information to matching results 11.match_add_link.py: Adds links to matching results Notes: Data in sample_data/ includes manually labeled samples. Output files in assets/ include the output of each manual/automated analysis referenced in the tables of the paper. Statistical Analysis (R) Performed via rq1_analysis/statistics.R Tests used: Chi-squared test (for distribution comparisons) Fisher’s exact test (for 2x2 tables) Krippendorff’s alpha (for inter-rater agreement) Tips Make sure paths are OS-consistent (Windows paths may need double-backslashes or raw strings). If Excel files don't open or save properly, check for open handles in Python. Use a virtual environment (venv) to manage dependencies cleanly. For any questions, please refer to the comments inside each script or contact the authors of the study.