RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices

<pre>We greatly appreciate your thoughtful feedback on our work! <br><br>The supplemented materials for Review #2570A is in the ''Response to Review #2570A.md''.<br>The supplemented materials for Review #2570B is in the ''Response to Review #2570B.md''.<br>The supplemented materials for Review #2570C is in the ''Response to Review #2570C.md''.<br><br><br></pre>This repository contains the implementation of **RealBench**, a comprehensive benchmark and evaluation framework for repository-level code aligned with real-world software development practices.<br><br>## 📝 Abstract<br><br>&gt; Repository-level code generation seeks to automate software development by generating complete codebases from specifications. While existing benchmarks like HumanEval evaluate LLMs on generating code from natural language requirements alone, they fail to reflect real-world industry practices where developers primarily work from structured system designs like UML diagrams. This gap means current benchmark scores may not accurately measure LLMs' practical utility. To address this, we propose RealBench, a novel repository-level benchmark aligned with real-world software development. RealBench provides both natural language requirements and UML diagrams (package and class diagrams) for 61 diverse, real-world Python repositories, along with comprehensive test suites. We evaluate six advanced LLMs (including GPT-4o, Claude, Gemini, DeepSeek, and Qwen) using three generation strategies: holistic, incremental, and retrieval-augmented generation (RAG). Our systematic evaluation, using five metrics across repository-level and class-level granularities, reveals key insights: LLMs perform poorly on this task (best Pass@1 is only 19.39%), struggle significantly with larger repositories, and while good at identifying modules from diagrams, often generate code with grammar and logic errors. The results show that holistic generation works best for small repositories, while incremental generation is superior for complex ones, and that detailed UML designs are crucial for performance. RealBench provides a standardized platform for evaluating and advancing LLMs for practical, design-driven code generation.<br><br>## 🔧 Features<br><br>- 🎯 **Comprehensive Benchmark**: Curated dataset of Python projects with UML diagrams, descriptions, and structure trees.<br>- 🤖 **Multi-Model Evaluation**: Support for GPT-4o, Claude, DeepSeek, Gemini, Qwen, and other state-of-the-art LLMs.<br>- 📊 **Multiple Approaches**: Evaluation of holistic, incremental, and RAG-based code generation methods.<br>- 🔍 **UML Analysis**: Advanced UML diagram parsing and analysis using Understand API.<br>- 📐 **Structure Preservation**: Maintains project structure and file organization during code generation.<br>- ⚙️ **Automated Evaluation**: Comprehensive evaluation pipeline with automated testing and metrics computation.<br><br>---<br><br>## 📄 Algorithm Supplement (Pseudocode)<br><br><br><br>## 📁 Project Structure<br><br>```text<br>.<br>├── datasets/ # Contains the benchmark datasets used for evaluation (Download from https://drive.google.com/drive/folders/16ZT8y-K-I6HN95wtv76xy5HGHNsaznQE?usp=sharing).<br>│ ├── all_proj/ # Complete project dataset with all files and dependencies.<br>│ └── projs_filtered_uml/ # Filtered dataset containing projects with UML diagrams.<br>│ ├── level1/ # Level 1 projects (basic complexity).<br>│ ├── level2/ # Level 2 projects (intermediate complexity).<br>│ ├── level3/ # Level 3 projects (advanced complexity).<br>│ └── level4/ # Level 4 projects (expert complexity).<br>├── src/ # Main source code for the RealBench framework.<br>│ ├── crawler/ # GitHub repository crawling and filtering.<br>│ │ ├── download.py # Downloads repositories from GitHub.<br>│ │ ├── filter.py # Filters repositories based on criteria.<br>│ │ └── github.py # GitHub API interaction utilities.<br>│ ├── uml_gen/ # UML diagram generation and analysis.<br>│ │ ├── file_uml.py # UML diagram generation using Understand API.<br>│ │ ├── tree.py # Project structure tree generation.<br>│ │ └── uml_dag.py # UML dependency graph analysis.<br>│ ├── model_gen/ # Code generation using various LLMs.<br>│ │ ├── generate/ # LLM inference implementations.<br>│ │ │ ├── gpt_inference.py # GPT model inference.<br>│ │ │ ├── claude_inference.py # Claude model inference.<br>│ │ │ ├── deepseek_inference.py # DeepSeek model inference.<br>│ │ │ ├── gemini_inference.py # Gemini model inference.<br>│ │ │ └── qwen_inference.py # Qwen model inference.<br>│ │ ├── make_prompts_UML/ # UML-based prompt generation.<br>│ │ │ ├── instruction_v3.txt # Instruction template for UML generation.<br>│ │ │ └── sys_gen_prom_con.py # System prompt construction.<br>│ │ ├── make_prompts_package/ # Package-level prompt generation.<br>│ │ ├── make_prompts_rag/ # RAG-based prompt generation.<br>│ │ └── results/ # Generated code results and evaluations (Download from https://drive.google.com/drive/folders/16ZT8y-K-I6HN95wtv76xy5HGHNsaznQE?usp=sharing).<br>│ │ ├── uml/ # UML-based generation results.<br>│ │ │ ├── holistic/ # Holistic generation results.<br>│ │ │ ├── incremental/ # Incremental generation results.<br>│ │ └── rag/ # RAG-based generation results.<br>│ └── cope_scripts/ # Data collection and processing scripts.<br>│ ├── collect_des.py # Collects project descriptions.<br>│ ├── parser_project.py # Parses project structure and dependencies.<br>│ └── generator_des.py # Generates project descriptions.<br>├── environment.yml # Conda environment configuration.<br>└── README.md # This file.<br>```<br><br>---<br><br>## ⚙️ Environment<br><br>* 🐧 Windows/Linux system (Windows 10/11 or Ubuntu 20.04+ recommended)<br>* 🐍 Python 3.9<br>* 📦 Conda environment management<br>* 🔧 Understand API (for UML analysis)<br><br>---<br><br><br><br>