SWE-bench_Lite_bm25_27K

### Dataset Summary
SWE-bench *Lite* is _subset_ of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.

The dataset was released as part of [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)

This dataset `SWE-bench_Lite_bm25_27K` includes a formatting of each instance using Pyserini's BM25 retrieval as described in the paper. The code context size limit is 27,000 `cl100k_base` tokens from the [`tiktoken`](https://github.com/openai/tiktoken) tokenization package used for OpenAI models.
The `text` column can be used directly with LMs to generate patch files.
Models are instructed to generate [`patch`](https://en.wikipedia.org/wiki/Patch_(Unix)) formatted file using the following template:
```diff
<patch>
diff
--- a/path/to/file.py
--- b/path/to/file.py
@@ -1,3 +1,3 @@
This is a test file.
-It contains several lines.
+It has been modified.
This is the third line.
</patch>
```

This format can be used directly with the [SWE-bench inference scripts](https://github.com/princeton-nlp/SWE-bench/tree/main/inference). Please refer to these scripts for more details on inference.

### 数据集概述
SWE-bench Lite 是 SWE-bench 的子集，后者是一款用于测试系统自动解决 GitHub 议题能力的数据集。该数据集从11个热门Python项目中采集了300组测试议题-拉取请求配对样本，评估通过单元测试验证完成，以拉取请求（PR）提交后的代码行为作为参考解决方案。

该数据集作为论文[SWE-bench：语言模型能否解决真实世界GitHub议题？](https://arxiv.org/abs/2310.06770)的配套资源正式发布。

本次发布的`SWE-bench_Lite_bm25_27K`数据集，按照论文所述的实现方式，使用Pyserini的BM25检索工具对每个实例进行了格式化处理。代码上下文的长度限制为27000个`cl100k_base` Token，该分词规则来自OpenAI模型所使用的[`tiktoken`](https://github.com/openai/tiktoken)分词工具包。

数据集中的`text`列可直接与大语言模型（Large Language Model，LLM）配合使用，用于生成补丁文件。模型需按照以下模板生成[`Patch`](https://en.wikipedia.org/wiki/Patch_(Unix))格式的补丁文件：
diff
<patch>
diff
--- a/path/to/file.py
--- b/path/to/file.py
@@ -1,3 +1,3 @@
This is a test file.
-It contains several lines.
+It has been modified.
This is the third line.
</patch>

该格式可直接与[SWE-bench推理脚本](https://github.com/princeton-nlp/SWE-bench/tree/main/inference)配合使用，如需了解推理环节的更多细节，请参考该脚本的相关文档。