Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

Replication Package Overview This replication package contains data and scripts for generating Software Bill of Materials (SBOMs) across multiple programming languages and analyzing them. The package is structured to support SBOM generation using different tools. Folder Structure 1. repo_details/ Contains CSV files listing repositories for each programming language. Files: {language}_repos.csv for each language (c, c#, c++, go, java, js, php, python, rust, swift)Content: Repository names in owner/repo format in the name columnPurpose: Source of truth for which repositories should have SBOMs generated 2. SBOM Output Folders Generated SBOM files organized by tool and language. gh_sboms/: SBOMs exported from GitHub's dependency graph API syft_sboms/: SBOMs generated using Syft tool trivy_sboms/: SBOMs generated using Trivy tool ms_sboms/: SBOMs generated using Microsoft SBOM tool Each folder has the structure: {tool}_sboms/├── c/├── c#/├── c++/├── go/├── java/├── js/├── php/├── python/├── rust/└── swift/ SBOM files follow the naming convention: {owner}:{repo}.json (e.g., torvalds:linux.json) 3. scripts/ Python scripts for SBOM generation and analysis. generate_gh_sboms.py: Exports SBOMs directly from GitHub's dependency graph APIgenerate_ms_trivy_syft_sboms.py: Generates SBOMs by cloning repositories and running local SBOM tools (Trivy, Syft, Microsoft SBOM tool) Usage Instructions 1. Exporting SBOMs from GitHub API Exports SBOMs directly from GitHub without cloning repositories. cd scripts/ python3 generate_gh_sboms.py Requirements: Python 3.6+ requests package (pip install requests) PyGithub package (pip install PyGithub) tqdm package (pip install tqdm) GitHub token with repo scope (https://github.com/settings/tokens) Setup Dependencies: pip install requests PyGithub tqdm Output: SBOMs in gh_sboms/ 2. Generating SBOMs with Tools (MS SBOM Tool, Trivy, Syft) Generates SBOMs using Trivy, Syft, and Microsoft SBOM tool by cloning repositories locally. cd scripts/ python3 generate_ms_trivy_syft_sboms.py Requirements: Git Trivy (trivy --version) Syft (syft --version) Microsoft SBOM tool (sbom-tool --version) GitHub token (optional, via GITHUB_TOKEN environment variable) Output: SBOMs in trivy_sboms/, syft_sboms/, and ms_sboms/ Workflow 1. Prepare Repository Data Ensure repo_details/*.csv files contain the repositories you want to analyze Each CSV should have a name column with owner/repo format entries 2. Generate SBOMs GitHub SBOM Tool: python3 scripts/generate_gh_sboms.py MS, Trivy, Syft: python3 scripts/generate_ms_trivy_syft_sboms.py 3. Data Analysis Use the generated SBOM files for analysis using the rest of the scripts Repository Name Format The package handles repository name normalization consistently: Input: genymobile/scrcpy (from CSV) Output: genymobile:scrcpy.json (SBOM filename) Troubleshooting repo_details directory not found: Ensure scripts are in replication_package/scripts/ Verify directory structure matches expected layout GitHub API rate limits: Use a GitHub token for higher rate limits Set GITHUB_TOKEN environment variable: export GITHUB_TOKEN=ghp_... Missing CSV columns: Ensure each CSV has a name column with owner/repo format entries Tool not found errors: For local tools: Install Trivy, Syft, and Microsoft SBOM tool For GitHub API: pip install requests PyGithub tqdm