A Systematic Evaluation of Large Language Models of Code

These are datasets for the paper:

"A Systematic Evaluation of Large Language Models of Code"

https://arxiv.org/pdf/2202.13169.pdf

The code is available at: https://github.com/VHellendoorn/Code-LMs

The file "unseen_test_sets.tar.gz" contains test sets of ~100 files in each of 12 programming languages.

These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them.

In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model.

The file "index.zip" includes an index of the training set file paths and commit SHAs.

The other files, such as "2-7B-150K.tar", are trained model checkpoints, as explained at https://github.com/VHellendoorn/Code-LMs .

本数据集配套论文《面向代码的大语言模型系统性评估》（A Systematic Evaluation of Large Language Models of Code），论文链接：https://arxiv.org/pdf/2202.13169.pdf。
代码开源地址：https://github.com/VHellendoorn/Code-LMs。

文件`unseen_test_sets.tar.gz`包含12种编程语言的测试集，每种语言对应约100个文件。
这些文件未被纳入The Pile数据集，因此GPT-Neo、GPT-J、GPT-NeoX等模型均未在该数据集上完成训练。
在论文中，我们借助该测试集对多款面向代码的语言模型展开对比评估，涵盖OpenAI的Codex、GPT-J、GPT-Neo、GPT-NeoX-20B、CodeParrot以及本团队自研的PolyCoder模型。

文件`index.zip`收录了训练集的文件路径与提交SHA哈希值（commit SHAs）的索引表。

其余文件（如`2-7B-150K.tar`）均为训练得到的模型检查点（checkpoint），详细说明可参考 https://github.com/VHellendoorn/Code-LMs。