Structural Dataset from Maven Central

This repository contains a compressed file with approximately 16.5 million POMs extracted from Maven Central. It serves as a structural dataset for Maven ecosystems and offers potential use cases such as the following: 💡 Analysing dependencies between artefacts to construct graphs or trees. 💡 Detecting vulnerabilities by cross-referencing <group_id>:<artifact_id>:<version> with databases such as Open Source Vulnerabilities (OSV) or the National Vulnerability Database (NVD). 💡 Building a local mirror of Maven for offline use. 💡 Indexing into a database for querying artefacts, versions, or licences. 💡 Studying the Java ecosystem by analysing trends and configurations in .pom files. 💡 Creating dependency recommendation systems based on real-world usage patterns. Folder Structure The file artifactory-index.tar.zst is 1.27 gigabytes in size and expands to a folder structure with a size of 85.8 gigabytes and 115 gibabytes. The folder structure follows the typical Maven Central path format to keep things intuitive. An example is shown below: dumps/ ├── org/apache/commons/lang3/3.17.0/ | └── commons-lang3-3.17.0.pom ├── org/apache/commons/lang3/3.16.0/ | └── commons-lang3-3.16.0.pom └── ... Disclaimer: On Unix operating systems you may have problems decompressing it due to the large number of files, which exceeds the number of inodes in the partitions. You must have a partition with more inodes than files. And on Windows operating systems you may have problems because of excessive length of some paths. You should disable the maximum path length limit. Updating the Folder Structure Lastly, the clone.py file contains a script written in Python 3.13 that updates and adds new .pom files to the folder structure. Dependencies must first be installed using the requirements.txt file. The clone.py script uses the maven_artifacts.txt file, where each line corresponds to an artefact indexed in Maven Central and is described in the format <group_id>:<artifact_id>:<version>. The current version of maven_artifacts.txt includes all artefacts indexed as of March 2025. Additional artefacts may be appended, or a different file with missing entries may be used; however, for the sake of reproducibility, it is advisable to make it available for others wishing to use this dataset. Processing POMs from a Large .tar File Without Extraction To efficiently process .pom files from a large Maven repository archive without consuming excessive disk space or exhausting inodes, you can mount the .tar archive as a virtual file system using ratarmount (Random Access To Archived Resources) in Ubuntu. This allows you to access and process individual files on demand without fully extracting the archive. First, ensure your system supports FUSE and has archivemount installed: pip install ratarmount. With ratarmount:- Creates an internal index of the .tar once. Then you can open, browse or read any file directly as if it were unzipped. It's fast and takes no extra space (except the .tar.index). If your archive is compressed, decompress it to obtain the raw .tar file use the command unzstd archive.tar.zst. Create a mount point for the archive with the command mkdir -p ~/maven_mount. Then mount the .tar file using the command ratarmount artifactory-index.tar maven_mount/, as I have uploaded the index created by ratarmount it should be mounted in a short time, if the tar is modified and the index has to be created again the approximate time is 30 minutes. The only needing is to decompress the index with command unzstd artifactory-index.tar.index.sqlite.zst. This mounts the archive as a read-only file system. Files are accessed on demand and not extracted to disk, significantly reducing inode usage. You can now traverse and process .pom files directly from the mount point. When processing is complete, unmount the archive with the command fusermount -u maven_mount. This approach is ideal when working with large Maven or Artifactory mirrors where full extraction is impractical. It provides selective, efficient access to metadata files while avoiding filesystem overhead. Deploy a JFrog OSS artifactory You can deploy on your local machine an artifactory based on the OSS version of JFrog using the docker compose up --build command. To configure JFrog Artifactory OSS, log in to http://localhost:8082/ using the default admin:password credentials and change the mandatory password. Then, in the Administration panel and the Repositories menu, create a local repository by selecting the Maven type and name it artifactory-index. Finally, update the PASSWORD constant in the upload.py script with your new password and run the script to load the data into Artifactory. Disclaimer: The script defined in upload.py is parallelized, how long it takes to load the artifactory depends on the available hardware, but it is obviously a time-consuming process and can take up a large amount of disk space due to the large number of files.