JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

JavaVulBench: A Java Vulnerability Detection Benchmark and Reproducibility Package JavaVulBench is a benchmark of 30,637 Java methods drawn from 1,740 CVEs. It provides five evaluation splits: random, project-disjoint, temporal, deduplicated, and CWE-family. It also includes a multi-backend evaluation harness covering fine-tuned encoders, local generative models, and API-served LLMs. This Zenodo record contains everything needed to reproduce the paper’s results from scratch. The package is split into two main archives so reviewers only download what they need. Files in This Record javavulbench-core.zip Size: approximately 490 MB compressed, approximately 780 MB on disk. This archive contains all code, configuration files, processed Java methods, the five evaluation splits, the 200-sample LLM evaluation probes, all shipped prediction files, the leakage audit, and one fine-tuned CodeBERT checkpoint. This archive is sufficient to run the Tier 1 verifier, which rebuilds the result tables in under one minute on CPU, and the Tier 1.5 verifier, which loads the included checkpoint and runs a forward pass in about one minute on CPU. javavulbench-full.tar Size: approximately 36 GB, uncompressed tar archive. This archive contains the raw mined CVE, commit, and source data, together with the remaining 13 fine-tuned encoder checkpoints. Extract this archive on top of the unpacked core archive only if you want to re-run the Tier 2 benchmark assembly pipeline or the Tier 3 full LLM evaluation, which requires an OpenRouter API key. SHA256SUMS This file contains checksums for both archives. Verify the downloaded files with sha256sum -c SHA256SUMS on Linux or macOS. On Windows PowerShell, use Get-FileHash with the SHA256 algorithm.