VulnGym

<p align="center"> <img src="./img/wukong_logo.png" alt="VulnGym" width="275"> </p> <h4 align="center"> <p> <a href="https://huggingface.co/datasets/tencent/VulnGym/blob/main/README_zh.md">中文</a> | <a href="#">English</a> </p> </h4> <p align="center"> <a href="https://github.com/Tencent/VulnGym"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-Tencent%2FVulnGym-181717?logo=github"></a> <a href="./LICENSE"><img alt="License" src="https://img.shields.io/badge/License-CC--BY--4.0-blue.svg"></a> <a href="#"><img alt="Version" src="https://img.shields.io/badge/version-0.1.2-green.svg"></a> </p> <p align="center"> <b>A Real-World, Project-Level Vulnerability Benchmark for White-Box Vulnerability-Hunting Agents</b> </p> **VulnGym** is a project-level benchmark for white-box vulnerability-hunting agents, designed to evaluate an agent's vulnerability detection capabilities within **real-world engineering contexts**, with **verifiable vulnerability trigger paths and code-semantic evidence chains**. **Three core design principles:** - **🏗️ Real project-level evaluation units** — every sample is bound to a specific vulnerable commit of a real repository, evaluating an agent's ability to discover and locate vulnerabilities inside real multi-file, multi-module engineering projects. - **🧠 Comprehensive vulnerability-type coverage** — the benchmark covers both business-logic defects that demand cross-module code-semantic reasoning (e.g., authorization bypass, broken authentication) and traditional security flaws (e.g., injection, path traversal), providing a comprehensive assessment of an agent's ability to discover diverse vulnerability classes. - **✅ Verifiable vulnerability paths** — each sample ships with a human-reviewed **reachable entry point** (`entry_point`), **critical operation** (`critical_operation`), and **cross-module reasoning chain** (`trace`), enabling reproducible, explainable, and deterministic evaluation. --- ## 📢 What's New - **2026-05-31** — 🔧 v0.1.2 data refresh: human-audited entries grew from **113 → 274 / 408 (67.2 %)**, covering **137 / 184 advisories (74.5 %)**. Additionally, `entry_point` / `critical_operation` / `trace` annotations were refined on 80 entries for improved accuracy. - **2026-05-17** — 🔧 v0.1.1 data refresh: added a `verify` field on every entry to mark human-audit status; **113 / 408 entries** (covering **61 / 184 advisories**) are now human-verified. Selected `entry_point` / `critical_operation` / `trace` values were also refined. - **2026-05-15** — 🎉 VulnGym v0.1.0 officially open-sourced! ## Table of Contents - [🔍 Why VulnGym](#-why-vulngym) - [✨ Dataset overview](#-dataset-overview) - [🗂️ Dataset structure](#️-dataset-structure) - [🚀 Quick start](#-quick-start) - [📊 Evaluating your tool](#-evaluating-your-tool) - [📦 Repository layout](#-repository-layout) - [📈 Baseline evaluation results](#-baseline-evaluation-results) - [📖 Citation](#-citation) - [🤝 Contribution Guide](#-contribution-guide) - [🙏 Acknowledgements](#-acknowledgements) - [📄 License](#-license) --- ## 🔍 Why VulnGym Existing vulnerability benchmarks have the following limitations when evaluating the real-world vulnerability-hunting capabilities of AI agents: | Limitation | Manifestation | |---|---| | **Insufficient evaluation granularity** | Most benchmarks use functions or diff snippets as the evaluation unit, failing to reflect an agent's ability to locate vulnerabilities within complete engineering projects | | **Narrow vulnerability types** | Over-emphasis on pattern-matchable CWE flaws such as SQL injection and buffer overflow, with little coverage of categories requiring deep contextual reasoning | | **Coarse-grained ground truth** | Typically binary labels (vulnerable / not vulnerable) or patch diffs, unable to precisely verify whether the agent locates the correct entry point and defect site | ## ✨ Dataset overview This is the **v0.1.2 release** of VulnGym. Data is provided as two JSONL files under the `data/` directory, exposed on the Hub as two configurations: - **`reports`** (`data/reports.jsonl`) — aggregated records at the GitHub Advisory granularity. - **`entries`** (`data/entries.jsonl`, *default*) — annotated records at the reachable entry point granularity. Each record contains `repo_url` and `commit`, allowing you to check out the full vulnerable source tree for the corresponding version. ### Data scale | Metric | Value | |---|---| | Advisories (reports) | **184** | | Reachable entry points (entries) | **408** | | Distinct projects | 38 | | Distinct repositories | 23 | | Human-audited entries (`verify = 1`) | **274 / 408 (67.2 %)** | | Human-audited advisories (≥ 1 verified entry) | **137 / 184 (74.5 %)** | ### Human audit status Starting in v0.1.1, every row in `entries` carries a `verify` field (`int`, `0` or `1`): - `verify == 1` — the entry's `entry_point`, `critical_operation`, and `trace` have been reviewed and confirmed by a human annotator. These rows form a high-confidence ground-truth subset and are recommended for strict, reproducible benchmarking. - `verify == 0` — automatically annotated; not yet human-confirmed. Useful for scale and recall studies, but values may still be refined in future releases. Of the **184** advisories, **108** have all of their entries verified and **29** are partially verified, for a total of **137** advisories with at least one human-audited entry. Future releases will continue to expand the verified subset. ### Vulnerability type distribution Every entry carries a two-level classification: `vuln_category_l1` (coarse type) and `vuln_category_l2` (fine-grained sub-type). **71.2 %** of advisories are business-logic vulnerabilities, classified with a **12-class + 1 fallback** taxonomy (see below). The remaining 28.8 % cover traditional vulnerability types. Full data model and field definitions are in [`SCHEMA.md`](SCHEMA.md). The initial release (v0.1.0) draws primarily from recent high-star open-source projects and focuses on frequently occurring business-logic vulnerabilities; future releases will continue expanding vulnerability categories and project coverage. > Note: one advisory may map to multiple entries — the counts below > are by **advisory (vulnerability)**, not by entry. **Business-logic advisories (131 / 184, 71.2 %) — `vuln_category_l2` breakdown:** | Sub-category | Advisories | % of BL | |---|---|---| | BL-AUTHZ-BROKEN — broken authorization logic | 31 | 23.7 % | | BL-AUTHZ-MISSING — missing authorization | 23 | 17.6 % | | BL-AGENT-CAPABILITY — AI / Agent capability boundary bypass | 20 | 15.3 % | | BL-PRIV-ESC — privilege escalation | 13 | 9.9 % | | BL-AUTH-BYPASS — authentication bypass | 11 | 8.4 % | <details> <summary>7 more sub-categories (33 advisories, 25.2 % of BL)</summary> | Sub-category | Advisories | % of BL | |---|---|---| | BL-ORIGIN-INTEGRITY — origin / signature / integrity check missing | 8 | 6.1 % | | BL-WORKFLOW-VIOLATION — workflow / state-machine violation | 7 | 5.3 % | | BL-INSECURE-DEFAULT — insecure default configuration | 6 | 4.6 % | | BL-RACE-LOGIC — business-layer race condition | 4 | 3.1 % | | BL-MULTI-TENANT — multi-tenant / isolation failure | 3 | 2.3 % | | BL-MASS-ASSIGNMENT — mass assignment / parameter pollution | 3 | 2.3 % | | BL-TRUST-BOUNDARY — implicit trust in internal input | 2 | 1.5 % | </details> <br> **Traditional vulnerability advisories (53 / 184, 28.8 %) — top `vuln_category_l1`:** | Category | Advisories | % of Trad. | |---|---|---| | Code Injection | 12 | 22.6 % | | Path Traversal / File ops | 9 | 17.0 % | | Command Injection | 8 | 15.1 % | | XSS | 5 | 9.4 % | | Sandbox Escape | 5 | 9.4 % | <details> <summary>4 more categories (14 advisories, 26.4 % of Trad.)</summary> | Category | Advisories | % of Trad. | |---|---|---| | SSRF | 4 | 7.5 % | | Authentication Bypass | 3 | 5.7 % | | Deserialization | 2 | 3.8 % | | Other (Template Injection, RCE, Supply Chain, etc.) | 5 | 9.4 % | </details> > Future releases will continue expanding vulnerability categories and project coverage. ## 🗂️ Dataset structure VulnGym is published as **two configurations** that share the join key `entries.report_id == reports.report_id`: ### Config: `entries` (default) — 408 rows One row per reachable entry point. | field | type | description | |---|---|---| | `entry_id` | `string` | Stable per-entry id, format `entry-{id:05d}` (e.g. `entry-00057`). | | `report_id` | `string` | GHSA id (upper-case) of the parent advisory. | | `source_link` | `string` | Canonical advisory URL. | | `vuln_ids` | `list[string]` | All known identifiers (CVE-* first, then GHSA-*). | | `origin` | `string` | Constant `"GitHub Advisory Database (reviewed)"`. | | `project` | `string` | Short project name (e.g. `open-webui`). | | `repo_url` | `string` | Source repository URL. | | `commit` | `string` | Vulnerable commit SHA — 40 lowercase hex chars. | | `vuln_title` | `string` | Per-entry title (may end with ` - <filename>` to disambiguate). | | `vuln_category_l1` | `string` | Coarse category (bilingual). | | `vuln_category_l2` | `string` | Sub-category (bilingual). | | `entry_point` | `dict` | Reachable entry — `{file, line, code}`. | | `critical_operation` | `dict` | Core defect location — `{file, line, code}`. | | `trace` | `list[dict]` | Ordered taint-flow steps — each `{file, line, code}`. | | `verify` | `int` | Human-audit flag (`1` = human-confirmed, `0` = auto-annotated). | ### Config: `reports` — 184 rows One row per GitHub Advisory, aggregating its entries. | field | type | description | |---|---|---| | `report_id` | `string` | GHSA id of the advisory. | | `source_link` | `string` | Advisory URL. | | `vuln_ids` | `list[string]` | Union of per-entry lists, normalized. | | `origin` | `string` | `"GitHub Advisory Database (reviewed)"`. | | `project` | `string` | Short project name. | | `repo_url` | `string` | Source repository URL. | | `commit` | `string` | Vulnerable commit SHA. | | `vuln_title` | `string` | Title with `" - filename"` suffix stripped. | | `num_entries` | `int` | Length of `entry_ids`. | | `entry_ids` | `list[string]` | All `entry_id`s of this report, sorted ascending. | > Full schema reference and invariants: see [`SCHEMA.md`](SCHEMA.md). ### Example row (`entries`) ```json { "entry_id": "entry-00057", "report_id": "GHSA-W7XJ-8FX7-WFCH", "source_link": "https://github.com/advisories/GHSA-w7xj-8fx7-wfch", "vuln_ids": ["CVE-2025-64495", "GHSA-W7XJ-8FX7-WFCH"], "origin": "GitHub Advisory Database (reviewed)", "project": "open-webui", "repo_url": "https://github.com/open-webui/open-webui", "commit": "9942de8011d4b5a141ac507c974c061c0cdad59a", "vuln_title": "Open WebUI Stored DOM XSS via Prompt Insertion Rich Text Feature", "vuln_category_l1": "XSS", "vuln_category_l2": "Stored XSS", "entry_point": { "file": "src/lib/components/chat/MessageInput/CommandSuggestionList.svelte", "line": 97, "code": "insertTextHandler(data.content);" }, "critical_operation": { "file": "src/lib/components/common/RichTextInput.svelte", "line": 348, "code": "tempDiv.innerHTML = htmlContent;" }, "trace": [ {"file": "…", "line": 42, "code": "…"} ], "verify": 1 } ``` ## 🚀 Quick start ### Load with 🤗 `datasets` ```python from datasets import load_dataset # entries config (default) — 408 reachable entry points entries = load_dataset("tencent/VulnGym", "entries", split="train") print(len(entries), "entries") print(entries[0]["entry_point"], "→", entries[0]["critical_operation"]) # reports config — 184 advisories reports = load_dataset("tencent/VulnGym", "reports", split="train") print(len(reports), "advisories") # High-confidence subset verified = entries.filter(lambda x: x["verify"] == 1) print(len(verified), "human-audited entries") # Filter by category xss = entries.filter(lambda x: x["vuln_category_l1"] == "XSS") print(len(xss), "XSS entries") ``` ### Load directly from JSONL ```python import json with open("data/entries.jsonl", encoding="utf-8") as f: entries = [json.loads(line) for line in f if line.strip()] ``` ### Pandas ```python import pandas as pd reports = pd.read_json("hf://datasets/tencent/VulnGym/data/reports.jsonl", lines=True) entries = pd.read_json("hf://datasets/tencent/VulnGym/data/entries.jsonl", lines=True) ``` ### Clone the repository ```bash git clone https://huggingface.co/datasets/tencent/VulnGym cd VulnGym python3 examples/load_dataset.py ``` ## 📊 Evaluating your tool Write your tool's findings to a JSONL file (one finding per line) and run: ```bash python3 examples/evaluate.py path/to/your_findings.jsonl -v ``` Each finding must carry at least `repo_url`, `commit`, `entry_point` (reachable entry point), and `critical_operation` (core defect location). `trace` (cross-module reasoning chain) is optional and ignored by the matcher. See `examples/example_result.jsonl` for a working sample. The script reports two metrics: - **Advisory-level recall** (primary) — `covered_advisories / usable_advisories`. An advisory is covered if **at least one** of its entries is matched. - **Entry-level recall** (secondary) — `matched_entries / usable_entries`. **Default matching policy** | Aspect | Default | |---|---| | Path match | normalized, exact | | Line tolerance | `\|Δline\| ≤ 5` on entry_point **and** critical_operation | | Direction | strict (entry_point-to-entry_point, critical_operation-to-critical_operation) | | `line == 0` in ground truth | excluded from numerator and denominator | All policies are documented and configurable via CLI arguments (`--line-tolerance`, etc.). > **Note:** The current evaluator **only computes recall / coverage** and > cannot penalize over-reporting. The resulting numbers should be > interpreted as coverage metrics, not a full precision-aware benchmark. ## 📦 Repository layout ``` VulnGym/ ├── README.md # this dataset card (English) ├── README_zh.md # 中文版 ├── SCHEMA.md # field reference & validation invariants ├── CHANGELOG.md ├── CITATION.cff ├── LICENSE # CC-BY-4.0 ├── data/ │ ├── reports.jsonl # 184 rows — one GitHub Advisory per row │ └── entries.jsonl # 408 rows — one entry point per row, with human-audit flag (verify) ├── examples/ │ ├── load_dataset.py # stdlib / pandas / HuggingFace datasets loader │ ├── example_result.jsonl # illustrative tool-findings submission │ └── evaluate.py # coverage / recall evaluator └── img/ └── wukong_logo.png ``` ## 📈 Baseline evaluation results > 🚧 **Coming soon** — We are systematically evaluating mainstream tools and AI agents. Results will be published alongside the technical report. ## 📖 Citation > 📚 **A companion paper is in preparation.** Until it is released, please cite VulnGym using the dataset entry below; we will update this section once the paper is publicly available. ```bibtex @misc{vulngym2026, title = {VulnGym: A Real-World, Project-Level Vulnerability Benchmark for White-Box Vulnerability-Hunting Agents}, author = {{Tencent Wukong Code Security Team and contributors}}, year = {2026}, version = {0.1.2}, howpublished = {\url{https://huggingface.co/datasets/tencent/VulnGym}}, note = {Dataset. A companion paper is in preparation; please check the repository for the latest citation.} } ``` Once the paper is public, the entry below will be filled in and should be preferred: ```bibtex @inproceedings{vulngym2026paper, title = {TBA — A companion paper for VulnGym is in preparation.}, author = {{To be announced}}, year = {TBA}, note = {Placeholder; will be replaced once the paper is publicly available.} } ``` See `CITATION.cff` for the machine-readable form. --- ## 🤝 Contribution Guide VulnGym aims to be an **open, reproducible, and continuously evolving** community benchmark. Contributions from both academia and industry are warmly welcomed: - 🧠 **Dataset contributions** — new advisories, additional reachable entry points for existing advisories, corrections to `entry_point` / `critical_operation` / `trace`. - 🔧 **Evaluator improvements** — precision / F1, per-category breakdowns, statistical significance (bootstrap CI), alternative matching policies. - 📊 **Evaluation result submissions** — submit your tool's evaluation results via PR to be included in the baseline comparison. - 💬 **Discussions & feedback** — file an [Issue](https://github.com/Tencent/VulnGym/issues) or start a [Discussion](https://github.com/Tencent/VulnGym/discussions) on the GitHub mirror. Please read `SCHEMA.md` before proposing data changes — all invariants listed there are enforced at release time. --- ## 🙏 Acknowledgements VulnGym is jointly built by the **Tencent Wukong Security Team** together with the following academic partners (listed in no particular order, final order TBD): - ARISE Lab, The Chinese University of Hong Kong - Systems Software & Security Lab, Fudan University - JC STEM Lab of Intelligent Cybersecurity, The University of Hong Kong - Narwhal-Lab, Peking University - Network Threat Analysis Lab, Institute of Information Engineering, Chinese Academy of Sciences Many thanks to all partners for their outstanding contributions to VulnGym. --- ## 📄 License The dataset is released under **CC-BY-4.0** — see [`LICENSE`](LICENSE). You may use it for commercial and academic purposes with attribution. Source code paths and commit hashes referenced in `entry_point` / `critical_operation` / `trace` fields belong to their respective upstream projects under their original licenses; consult the referenced repositories before reusing any quoted code fragment.