ABC-Bench

<div align="center"> <h1 style="font-size: 40px; line-height: 1.1; margin: 0;"> ABC-Bench </h1> </div> <div align="center"> 💻 <a href="https://github.com/OpenMOSS/ABC-Bench"><b>Code</b></a>&nbsp&nbsp | &nbsp&nbsp 📑 <a href="https://arxiv.org/abs/2601.11077"><b>Paper</b></a>&nbsp&nbsp | &nbsp&nbsp 📝 <a href="https://dawning-road.github.io/blog/abc-bench"><b>Blog</b></a>&nbsp&nbsp </div> # 📖 Overview **ABC-Bench** is a benchmark for **A**gentic **B**ackend **C**oding. It evaluates whether code agents can **explore real repositories**, **edit code**, **configure environments**, **deploy containerized services**, and **pass external end-to-end API tests** (HTTP-based integration tests) across realistic backend stacks. # 📊 Benchmark Composition  # 🚀 Why ABC-Bench? - **End-to-End Lifecycle**: repository exploration → code editing/implementation → environment setup → containerized deployment → **external end-to-end API verification**. - **Real-World Diversity**: **224 tasks** curated from **127 MIT-licensed repositories**, spanning **8 languages** and **19 frameworks**. - **Environment-Aware Tasks**: **92 tasks** require autonomous environment configuration and containerized service startup. - **Automated Construction**: built via **ABC-Pipeline** with minimal manual intervention, enabling scalable task creation and future expansions. - **Challenging Baselines**: even state-of-the-art models remain far from fully reliable. # 📂 Using the Dataset ## 1. Download (Git LFS Required) ```bash # 1. Install Git LFS git lfs install # 2. Clone the repository git clone [e-mail]:datasets/OpenMOSS-Team/ABC-Bench # 3. Enter the repository directory cd ABC-Bench # 4. Ensure large files are fully downloaded git lfs pull ``` ## 2. Extract ```bash tar -xzf tasks.tar.gz ``` ## 3. Directory Structure After extraction, the tasks directory will look like this: ```text tasks/ ├── task_<repo_slug>__<scenario_name> │ ├── <repo_name> # repo source code │ ├── task.yaml # instruction + metadata │ ├── run-tests.sh # HTTP-level validator driven by pytest │ ├── docker-compose.yaml │ ├── Dockerfile │ ├── solution.sh │ └── tests/ └── ... (224 total task folders) ``` Please refer to our [Quickstart](https://github.com/OpenMOSS/ABC-Bench) guide to proceed with the evaluation. # 📝 Citation If you use **ABC-Bench Tasks** in your research, please cite the paper: ```bibtex @misc{yang2026abcbenchbenchmarkingagenticbackend, title={ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development}, author={Jie Yang and Honglin Guo and Li Ji and Jiazheng Zhou and Rui Zheng and Zhikai Lei and Shuo Zhang and Zhiheng Xi and Shichun Liu and Yuxin Wang and Bo Wang and Yining Zheng and Tao Gui and Xipeng Qiu}, year={2026}, eprint={2601.11077}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2601.11077}, } ``` # ⚖️ License This dataset card is released under the **odc-by**.