Schwerli/SWE-bench_Pro

SWE-Bench Pro是一个具有挑战性的企业级数据集，用于测试代理在长期软件工程任务中的能力。数据集结构遵循SWE-Bench Verified，并包含多个额外字段。数据字段包括仓库标识符、实例ID、基础提交哈希、代码补丁、测试补丁、问题描述、项目需求、接口规范、仓库语言、测试用例等。

SWE-Bench Pro is a challenging, enterprise-level dataset for testing agent ability on long-horizon software engineering tasks. The dataset structure follows SWE-Bench Verified with several extra fields. Data fields include repo (repository identifier), instance_id (unique identifier for each instance), base_commit (git commit hash of the base version), patch (the golden code patch/diff), test_patch (test cases related to the patch), problem_statement (description of the issue being addressed), requirements (project requirements or dependencies), interface (API or interface specifications), repo_language (programming language of the repository), fail_to_pass (test cases that should pass after patch application), pass_to_pass (test cases that should continue passing), issue_specificity (specificity of the issue), issue_categories (categories or tags for the issue type), before_repo_set_cmd (repo set command for testing), and selected_test_files_to_run (files selected for testing).

搜集汇总

背景与挑战

背景概述

SWE-Bench Pro是一个企业级软件工程评估数据集，包含731个实例，覆盖多个开源项目（如NodeBB、qutebrowser、ansible）的代码补丁、测试用例和问题描述，用于测试AI代理在长期、复杂软件任务上的能力。数据集遵循SWE-Bench Verified结构，并扩展了额外字段，支持对代码修复、功能实现等任务的评估。