ISSTA 2026 Artifact Package for Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

This Zenodo record contains the ISSTA 2026 Artifact Evaluation package for the paper Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering. Kick-the-tires update: this version adds a single reviewer convenience archive, issta26-fp-artifact-complete.zip. This is now the recommended download path. The archive contains the Docker image archive, source archive, full core evidence data archive, checksums, release manifest, license, and all reviewer-facing documentation. The package is designed around the artifact-evaluation path requested by the ISSTA 2026 AE call: reviewers load a Docker image, unpack the core evidence data archive, and run offline scripts that recompute and check the paper's hard numerical claims. The primary validation path does not require API keys, internet access during experiments, commercial LLM services, CodeQL/Joern/SAST reruns, or SWE-agent reruns. Storage Estimate Downloaded issta26-fp-artifact-complete.zip: about 1.6 GiB. Unzipped release directory: about 1.6 GiB. Core evidence archive artifact-data-core.tar.gz: about 1.5 GiB. Uncompressed/extracted artifact-data/: about 12 GiB on disk, with about 11 GiB apparent size. Docker image after docker load: about 155 MiB. Peak local space during validation, if the downloaded zip, unzipped release directory, data archive, and extracted data are all kept: about 17 GiB. Recommended free space before starting: at least 25 GiB. Documentation Map README.md: start here for download, unpacking, Docker loading, smoke-test, and full verification commands. ARTIFACT_README.md: expanded artifact guide, including supported claims, unsupported optional reruns, and per-script workflow. CLAIM_REPRODUCTION_MATRIX.md: maps paper claims to evidence directories and reproduction scripts. AE_REQUIREMENT_AUDIT.md: checklist against the ISSTA 2026 Artifact Evaluation requirements. DATA_LAYOUT.md: explains the mounted artifact-data/ evidence tree used by the Docker container. BENCHMARKJAVA.md: explains OWASP Benchmark for Java v1.2, the local BenchmarkJavaModified name, and the RQ1 SAST-baseline stage. REQUIREMENTS.md, STATUS.md, LICENSE, and OPTIONAL_RERUNS.md: platform requirements, badge status, provenance/licensing, and non-required online rerun notes. Files issta26-fp-artifact-complete.zip: single-file artifact package for reviewers. issta26-fp-artifact-complete.zip.sha256: SHA-256 checksum for the complete archive. artifact-data-core.tar.gz: separate full core evidence data archive, provided as an additional convenience file for users who prefer individual downloads. Inside issta26-fp-artifact-complete.zip, the directory issta26-fp-artifact-release/ contains: README.md: quick-start instructions. BENCHMARKJAVA.md: explanation of OWASP Benchmark for Java v1.2, the local BenchmarkJavaModified working-copy name, and the two RQ1 stages: SAST baseline construction and agent filtering. ARTIFACT_README.md, REQUIREMENTS.md, STATUS.md, DATA_LAYOUT.md, AE_REQUIREMENT_AUDIT.md, and CLAIM_REPRODUCTION_MATRIX.md: reviewer-facing documentation. issta26-fp-artifact-image.tar.gz: Docker image archive for the offline verification environment. artifact-source.tar.gz: source code, Dockerfile, verification scripts, manifests, and artifact documentation. artifact-data-core.tar.gz: core evidence data archive. artifact-data-core.tar.gz.sha256: checksum for the core evidence data archive. SHA256SUMS and RELEASE_MANIFEST.json: release integrity and provenance metadata. Data Archive Unpacking The ISSTA AE page recommends common archive formats such as .zip or .tar.gz. The complete artifact is therefore provided as .zip, and the core evidence data inside it is an ordinary .tar.gz archive. Unpack it with: unzip issta26-fp-artifact-complete.zip cd issta26-fp-artifact-release sha256sum -c SHA256SUMS sha256sum -c artifact-data-core.tar.gz.sha256 tar -xzf artifact-data-core.tar.gz Offline Validation After loading the Docker image and unpacking the evidence data, reviewers can run: docker load -i issta26-fp-artifact-image.tar.gz docker run --rm -v "$PWD/artifact-data:/artifact/data:ro" \ -v "$PWD/artifact-output:/artifact/out" \ issta26-fp-artifact:ae /artifact/scripts/check_artifact.sh docker run --rm -v "$PWD/artifact-data:/artifact/data:ro" \ -v "$PWD/artifact-output:/artifact/out" \ issta26-fp-artifact:ae /artifact/scripts/verify_claims.sh --skip-soft The release packaged inside the complete archive was locally validated before upload. The strict hard-claim check completed with All hard claim checks passed. Scope The functional artifact path verifies the saved evidence for the paper's claims, including the OWASP Benchmark false-positive filtering results, Vul4J results, OSS-Fuzz generalization results, true-positive retention analysis, contamination probe, and manually audited qualitative RQ4 evidence. Full online reruns of LLM agents are documented as optional extended reproduction because they require external model services and, for some agents, additional runtime infrastructure.