microsoft/FEA-Bench

FEA-Bench是一个用于评估仓库级增量代码开发能力的基准测试数据集。它包含一个测试集，其中包括来自83个GitHub仓库的1,401个任务实例，主要关注新功能的实现。该数据集由FEA-Bench论文的作者整理，旨在评估LLMs在仓库级代码开发方面的性能。数据集中不包含任何个人或敏感信息，并且不适用于LLMs的训练，以避免污染。README文件还列出了所有涉及的GitHub仓库及其相应的许可证。

The FEA-Bench is a benchmark designed to evaluate the capabilities of repository-level incremental code development. It includes a test set with 1,401 task instances from 83 GitHub repositories, focusing on new feature implementation. The dataset is curated by the authors of the FEA-Bench paper and is intended for evaluating the performance of LLMs on repository-level code development. The dataset does not include any personal or sensitive information and is not intended for training LLMs to avoid contamination. The README also includes a list of all involved GitHub repositories and their respective licenses.