SWE-Cycle

# SWE-Cycle **SWE-Cycle** is a comprehensive benchmark for evaluating AI coding agents on full software engineering lifecycle tasks. It decomposes the software development process into four isolated phases and one integrated end-to-end task: - **Environment Setup (Env)**: Configure the development environment from scratch - **Code Implementation (CodeImpl)**: Fix bugs or implement features given a configured environment - **Test Generation (TestGen)**: Write tests that capture the intended behavior - **Full Cycle (FullCycle)**: Complete all three phases in a single end-to-end task ## Dataset Overview | Source | Records | Languages | |--------|---------|-----------| | SWE-bench Verified | 225 | Python | | SWE-bench Pro | 203 | Python, Go, TypeScript, JavaScript | | Multilingual | 61 | Java, Ruby, C, PHP, Rust, Go, JavaScript | | **Total** | **489** | **9 languages** | ## Language Distribution | Language | Count | |----------|-------| | Python | 337 | | Go | 88 | | C | 16 | | Ruby | 13 | | JavaScript | 12 | | Rust | 7 | | TypeScript | 6 | | Java | 6 | | PHP | 4 | ## Fields | Field | Description | |-------|-------------| | `instance_id` | Unique identifier for each instance | | `repo` | GitHub repository (owner/name) | | `base_commit` | Base commit hash | | `version` | Version identifier | | `patch` | Gold code patch (solution) | | `test_patch` | Gold test patch | | `problem_statement` | Natural language problem description | | `hints_text` | Optional hints | | `FAIL_TO_PASS` | Tests that should pass after fix (JSON string) | | `PASS_TO_PASS` | Regression tests (JSON string) | | `created_at` | Issue creation timestamp | | `difficulty` | Difficulty level (SWE-bench Verified only) | | `environment_setup_commit` | Environment setup commit (SWE-bench Verified only) | | `run_script` | Per-instance run script (SWE-bench Pro only) | | `parsing_script` | Per-instance parser script (SWE-bench Pro only) | | `before_repo_set_cmd` | Repo initialization command (SWE-bench Pro only) | | `selected_test_files_to_run` | Test files to execute (SWE-bench Pro only) | | `image_name` | Docker image name (SWE-bench Pro only) | | `source` | Data source: `swebench_verified`, `swebench_pro`, or `multilingual` | | `language` | Programming language of the repository | ## Usage ```python from datasets import load_dataset ds = load_dataset("tubehhh/SWE-Cycle", split="test") ``` ## Evaluation SWE-Cycle employs a dual-track evaluation system: 1. **Static evaluation**: Script-based grading using test execution results 2. **Dynamic evaluation**: LLM-as-judge evaluation for semantic correctness The final score combines both tracks, providing robust assessment of agent capabilities across all phases of the software development lifecycle. ## Citation ```bibtex @inproceedings{swe-cycle2026, title={SWE-Cycle: A Multi-Phase Benchmark for Evaluating AI Coding Agents on Full Software Engineering Lifecycles}, author={Guan, Hao}, year={2026} } ``` ## License This dataset is released under the [MIT License](https://opensource.org/licenses/MIT).