sourcegraph/code-multi-line-infilling-benchmark

--- dataset_info: features: - name: repo dtype: string - name: instance_id dtype: string - name: base_commit dtype: string - name: file_path dtype: string - name: prefix dtype: string - name: suffix dtype: string - name: middle dtype: string - name: context list: - name: content dtype: string - name: file_path dtype: string - name: rank dtype: int64 - name: patch dtype: string - name: test_patch dtype: string - name: problem_statement dtype: string - name: hints_text dtype: string - name: created_at dtype: string - name: version dtype: string - name: FAIL_TO_PASS dtype: string - name: PASS_TO_PASS dtype: string - name: environment_setup_commit dtype: string splits: - name: test num_bytes: 34758248 num_examples: 375 download_size: 12190963 dataset_size: 34758248 configs: - config_name: default data_files: - split: test path: data/test-* --- ### Dataset Summary This dataset is used to evaluate Multi-Line fill in the middle code completion capabilities of a system. The dataset is derived from [SWE-Bench dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench). Evaluation is performed by stiching the generated middle portion, with the other patch and passing into the SWE Evaluation harness, which runs unit test verification and calculate Pass@1. ### Data Instances In addition to the fields already calculated by [SWE-Bench dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench), this dataset contains five additional fields. Additional Fields: ``` file_path: (str) - Path of the file relative of the root of the repo, in which to complete the multi-line code. prefix: (str) - Code before the cursor position (ideally at this point, the code completion is triggered). suffix: (str) - Code after the cursor position. middle: (str) - The original ground truth code in the file. context: (List[Dict]) - The list of Dict, sampled from the other files in the repo using BM25 in addition to Sematic chunking. ``` Note: Each item in the `context` is a dictionary, containing the following fields: ``` file_path: (str) - The path of the file, from which the context is extracted. content: (str) - Code snippet from the file to be used as a context. rank: (str) - The rank of the snippet. ``` A combination of `prefix`, `suffix`, `file_path` and `context` is used to make the final prompt. For the sake of completeness, we list existing SWE-Bench dataset fields: ``` instance_id: (str) - A formatted instance identifier, usually as repo_owner__repo_name-PR-number. patch: (str) - The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue. repo: (str) - The repository owner/name identifier from GitHub. base_commit: (str) - The commit hash of the repository representing the HEAD of the repository before the solution PR is applied. hints_text: (str) - Comments made on the issue prior to the creation of the solution PR’s first commit creation date. created_at: (str) - The creation date of the pull request. test_patch: (str) - A test-file patch that was contributed by the solution PR. problem_statement: (str) - The issue title and body. version: (str) - Installation version to use for running evaluation. environment_setup_commit: (str) - commit hash to use for environment setup and installation. FAIL_TO_PASS: (str) - A json list of strings that represent the set of tests resolved by the PR and tied to the issue resolution. PASS_TO_PASS: (str) - A json list of strings that represent tests that should pass before and after the PR application. ```数据集信息： 特征： - 字段名：repo，数据类型：字符串 - 字段名：instance_id，数据类型：字符串 - 字段名：base_commit，数据类型：字符串 - 字段名：file_path，数据类型：字符串 - 字段名：prefix，数据类型：字符串 - 字段名：suffix，数据类型：字符串 - 字段名：middle，数据类型：字符串 - 字段名：context，数据类型：列表，列表元素包含以下子字段： - 字段名：content，数据类型：字符串 - 字段名：file_path，数据类型：字符串 - 字段名：rank，数据类型：64位整数 - 字段名：patch，数据类型：字符串 - 字段名：test_patch，数据类型：字符串 - 字段名：problem_statement，数据类型：字符串 - 字段名：hints_text，数据类型：字符串 - 字段名：created_at，数据类型：字符串 - 字段名：version，数据类型：字符串 - 字段名：FAIL_TO_PASS，数据类型：字符串 - 字段名：PASS_TO_PASS，数据类型：字符串 - 字段名：environment_setup_commit，数据类型：字符串 划分集： - 划分名称：test，字节数：34758248，样本数量：375 下载大小：12190963，数据集总大小：34758248 配置： - 配置名称：default，数据文件： - 划分：test，路径：data/test-* ### 数据集摘要 本数据集用于评估系统的多行中间代码补全能力。该数据集源自[SWE-Bench数据集（SWE-Bench dataset）](https://huggingface.co/datasets/princeton-nlp/SWE-bench)。评估流程为将生成的中间代码片段与其余补丁拼接后，输入至SWE评估工具链，该工具链将运行单元测试验证并计算Pass@1指标。 ### 数据实例 除[SWE-Bench数据集（SWE-Bench dataset）](https://huggingface.co/datasets/princeton-nlp/SWE-bench)已涵盖的字段外，本数据集额外新增5个字段。 新增字段说明： file_path：（字符串类型）—— 需执行多行代码补全的文件相对于仓库根目录的路径。 prefix：（字符串类型）—— 光标位置前的代码（理想场景下，此处即为代码补全的触发点）。 suffix：（字符串类型）—— 光标位置后的代码。 middle：（字符串类型）—— 文件中原始的基准事实（ground truth）代码。 context：（字典列表）—— 通过BM25算法结合语义分块，从仓库内其他文件中采样得到的字典列表。 注意：`context`中的每个元素均为字典，包含以下字段： file_path：（字符串类型）—— 提取上下文的文件路径。 content：（字符串类型）—— 用作上下文的代码片段。 rank：（字符串类型）—— 该代码片段的排序位次。 最终的提示词由`prefix`、`suffix`、`file_path`以及`context`组合构建而成。 为保证完整性，此处列出SWE-Bench数据集原有的全部字段： instance_id：（字符串类型）—— 格式化的实例标识符，通常采用repo_owner__repo_name-PR-number的格式。 patch：（字符串类型）—— 金标准补丁，即由拉取请求（PR）生成的、解决对应问题的补丁（不含测试相关代码）。 repo：（字符串类型）—— GitHub上的仓库所有者与名称标识符。 base_commit：（字符串类型）—— 应用解决方案拉取请求前，代表仓库HEAD状态的提交哈希值。 hints_text：（字符串类型）—— 解决方案拉取请求首次提交创建日期之前，在对应议题下留下的评论内容。 created_at：（字符串类型）—— 拉取请求的创建日期。 test_patch：（字符串类型）—— 由解决方案拉取请求贡献的测试文件补丁。 problem_statement：（字符串类型）—— 对应议题的标题与正文内容。 version：（字符串类型）—— 运行评估时需使用的环境安装版本。 environment_setup_commit：（字符串类型）—— 用于环境搭建与配置的提交哈希值。 FAIL_TO_PASS：（字符串类型）—— 由JSON格式字符串组成的列表，代表通过该拉取请求解决且与议题解决相关的测试集合。 PASS_TO_PASS：（字符串类型）—— 由JSON格式字符串组成的列表，代表在应用拉取请求前后均应通过的测试集。