JetBrains/git_good_bench

GitGoodBench Lite是一个包含900个样本的子集，用于评估AI代理在解决git任务（见支持的场景）方面的性能。数据集中的样本均匀地分为编程语言Python、Java和Kotlin以及样本类型合并冲突解决和文件提交链。该数据集还包含来自479个独特的、开源的GitHub仓库的数据，这些仓库具有宽松的许可证，拥有至少1000个星级、5个分支和10个贡献者，并且不是分叉或存档的仓库。数据集包含两种类型的样本：合并和文件提交链。合并场景包含一个或多个合并冲突，所有合并冲突都保证在Python、Java或Kotlin文件中。文件提交链场景由两个提交组成，涵盖连续的提交，用于评估代理创建有意义、连贯的提交或通过变基改进本地树的能力。

GitGoodBench Lite is a subset of 900 samples for evaluating the performance of AI agents in resolving git tasks (see Supported Scenarios). The samples in the dataset are evenly split across the programming languages Python, Java, and Kotlin, as well as the sample types merge conflict resolution and file-commit chain. The dataset contains data from 479 unique, open-source GitHub repositories with permissive licenses that have at least 1000 stars, 5 branches, and 10 contributors and are not forks or archived. The dataset includes two types of samples: merge and file_commit_chain. Merge scenarios contain one or more merge conflicts in Python, Java, or Kotlin files. File-commit chain scenarios consist of two commits, covering consecutive commits, used to evaluate the agents ability to create meaningful, cohesive commits or improve the local tree through rebasing.