RepoBench Dataset

RepoBench is a benchmark designed for evaluating repository-level code auto-completion systems, focusing on more complex, real-world programming scenarios involving multiple files. It comprises three tasks: RepoBench-R (Retrieval), measuring the system's ability to retrieve relevant code snippets; RepoBench-C (Code Completion), assessing the prediction of the next line of code with both in-file and cross-file context; and RepoBench-P (Pipeline), evaluating complex tasks requiring both retrieval and prediction. RepoBench aims to provide a comprehensive performance comparison to foster continuous improvement in auto-completion systems.

RepoBench是一款专为评估代码库层面自动补全系统设计的基准测试，其核心在于针对涉及多个文件、更加复杂且贴近实际的编程场景。该基准测试包含三项任务：RepoBench-R（检索）任务，旨在衡量系统检索相关代码片段的能力；RepoBench-C（代码补全）任务，评估系统在文件内及跨文件上下文中预测下一行代码的能力；以及RepoBench-P（管道）任务，评估需要同时进行检索和预测的复杂任务。RepoBench的目标是提供全面性能比较，以促进自动补全系统的持续优化。