ByteDance/PatchEval

PatchEval是一个基准测试，旨在系统评估LLMs和代理在自动漏洞修复任务中的表现。它包含1,000个来自2015年至2025年报告的CVE漏洞，覆盖65个CWE类别，涉及Go、JavaScript和Python三种编程语言。其中230个CVE配备了Docker化的沙盒环境，支持通过概念验证（PoC）和单元测试进行运行时补丁验证。每个漏洞实例都是一个JSON对象，包含CVE ID、描述、CWE信息、仓库URL、补丁URL、编程语言、漏洞代码片段、修复代码片段、补丁差异、PoC测试命令和单元测试命令等信息。

PatchEval is a benchmark designed to systematically evaluate LLMs and Agents in the task of automated vulnerability repair. It includes 1,000 vulnerabilities sourced from CVEs reported between 2015 and 2025, covering 65 CWE categories across Go, JavaScript, and Python. A subset of 230 CVEs is paired with Dockerized sandbox environments that enable runtime patch validation through Proof-of-Concept (PoC) and unit testing. Each vulnerability in the PatchEval dataset is a JSON object with fields such as CVE ID, description, CWE info, repository URL, patch URLs, programming language, vulnerable code snippets, fixed code snippets, patch diff, PoC test command, and unit test command.