The Software Heritage Graph Dataset

Software Heritage is the largest existing public archive of software source<br> code and accompanying development history: it currently spans more than five<br> billion unique source code files and one billion unique commits, coming from<br> more than 80 million software projects. This is the Software Heritage graph dataset: a fully-deduplicated<br> Merkle DAG representation of the Software Heritage archive. The dataset links<br> together file content identifiers, source code directories, Version Control<br> System (VCS) commits tracking evolution over time, up to the full states of VCS<br> repositories as observed by Software Heritage during periodic crawls. The<br> dataset’s contents come from major development forges (including GitHub and<br> GitLab), FOSS distributions (e.g., Debian), and language-specific package<br> managers (e.g., PyPI). Crawling information is also included, providing<br> timestamps about when and where all archived source code artifacts have been<br> observed in the wild. The Software Heritage graph dataset is available in multiple formats, including<br> downloadable CSV dumps and Apache Parquet files for local use, as well as a<br> public instance on Amazon Athena interactive query service for ready-to-use<br> powerful analytical processing. By accessing the dataset, you agree with the Software Heritage Ethical Charter<br> for using the archive data, and the terms of use for bulk access. If you use this dataset for research purposes, please cite the following paper: Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli. <br> <em>The Software Heritage Graph Dataset: Public software development under one roof</em>. <br> In proceedings of MSR 2019: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Co-located with ICSE 2019. <br> preprint, bibtex You can also refer to the above paper for more information the dataset and sample queries.