A Systematic Evaluation of Large Language Models of Code

These are datasets for the paper: "A Systematic Evaluation of Large Language Models of Code" https://arxiv.org/pdf/2202.13169.pdf The code is available at: https://github.com/VHellendoorn/Code-LMs The file "unseen_test_sets.tar.gz" contains test sets of ~100 files in each of 12 programming languages. These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them. In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model. The file "index.zip" includes an index of the <strong>training set</strong> file paths and commit SHAs.