macrocosm-os/code-parrot-github-code

GitHub Code数据集包含来自GitHub的1.15亿个代码文件，涵盖32种编程语言和60种文件扩展名，总数据量达到1TB。该数据集是从Google BigQuery上的公共GitHub数据集中创建的。数据集中的每个文件都包含代码内容、仓库名称、文件路径、编程语言、许可证类型和文件大小等信息。数据集主要用于文本生成任务，特别是语言建模。

The GitHub Code dataset consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in 1TB of data. The dataset was created from the public GitHub dataset on Google BigQuery. Each file in the dataset includes the code content, repository name, file path, programming language, license type, and file size. The dataset is primarily used for text generation tasks, particularly language modeling.