CSJianYang/ExecRepoBench

在这项工作中，我们引入了一个新颖的框架，通过创建仓库级别的基准ExecRepoBench和指令语料库Repo-Instruct，旨在提高开源大语言模型（LLMs）在涉及多个文件之间复杂依赖关系的实际编码场景中的功能。ExecRepoBench包括来自活跃Python仓库的1.2K个样本。此外，我们提出了一种基于多级语法的补全方法，该方法基于抽象语法树来屏蔽各种逻辑单元（如语句、表达式和函数）中的代码片段。

ExecRepoBench is a novel framework for enhancing code completion in software development, aiming to improve the functionality of open-source large language models (LLMs) in real-world coding scenarios involving complex interdependencies across multiple files. The dataset includes 1.2K samples from active Python repositories and employs a multi-level grammar-based completion methodology conditioned on the abstract syntax tree to mask code fragments at various logical units for evaluating code completion.