Dataset and code used in Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

Warning: the whole dataset occupies 10.4 GB (11,228,158,798 bytes) in total. We recommend that you reserve enough space if you want to decompress it. Artifact: Sifting the Noise — A Comparative Study of LLM Agents in Vulnerability False Positive Filtering This repository contains the replication package for the paper: Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering Overview This artifact provides the complete experimental data, evaluation framework, and results for our comparative study of LLM-based agent frameworks (Aider, OpenHands, SWE-agent) and vanilla LLM prompting for filtering false positives (FPs) generated by Static Application Security Testing (SAST) tools. Repository Structure artifacts/├── README.md # This file├── Evaluation Framework/ # Scripts to run experiments├── Initial SAST Scan Result/ # Raw SAST tool outputs├── RQ1/ # RQ1: FP filtering on OWASP Benchmark├── RQ2/ # RQ2: Real-world Vul4J alerts└── Discussion (TP Retention)/ # TP retention analysis Folder Descriptions Folder Description Evaluation Framework Core scripts for running LLM agents and vanilla prompting on triage tasks Initial SAST Scan Result Raw outputs from CodeQL, Semgrep, Joern, and SonarQube on OWASP Benchmark RQ1 Task definitions and results for RQ1 — FP filtering effectiveness on OWASP Benchmark RQ2 Task definitions and results for RQ2 — Real-world vulnerability triage on Vul4J Discussion (TP Retention) Data for analyzing TP retention (avoiding false negatives) Research Questions RQ1: How effective are different LLM-based agent frameworks in filtering false positives generated by SAST tools? RQ2: How do LLM-based agents perform on real-world vulnerability alerts from open-source Java projects? RQ3: What are the key success drivers and recurring failure modes of LLM-based agents in FP identification? Task Folders (Inputs) The following folders contain the triage task definitions (JSON case files) used as inputs to run_batch_triage.py: Task Folder Location Description triage-owasp-benchmark RQ1/ False positive cases from OWASP Benchmark triage-vul4j-alerts RQ2/ Real-world CodeQL alerts from Vul4J projects triage-owasp-benchmark-tp Discussion (TP Retention)/ True positive cases for retention analysis Experimental Configuration Agent Frameworks Aider — AI pair programming assistant OpenHands — Autonomous coding agent platform SWE-agent — Software engineering agent for repository-level tasks Vanilla LLM — Zero-shot prompting baseline Backbone Models Claude Sonnet 4 (claude-sonnet-4-20250514) DeepSeek Chat (deepseek_deepseek-chat) GPT-5 (gpt-5, gpt-5-2025-08-07) Datasets OWASP Benchmark v1.2: 2,740 test cases (1,415 TP, 1,325 FP) across 11 CWE categories Vul4J: Real-world Java vulnerabilities from open-source projects Getting Started Review task definitions: Examine JSON files in the triage-* folders Run experiments: Use scripts in Evaluation Framework/ to execute agents Analyze results: Check runs/ folders for agent outputs and verdicts See individual folder READMEs for detailed documentation. Output Format All agents produce results in a standardized JSON format: { "label": "TP|FP|UNKNOWN", "confidence": "high|medium|low", "confidence_score": 0.0, "reason": "explanation grounded in code evidence", "evidence": [...], "agent": "agent-name", "case_id": "case-identifier"}