nyuuzyou/gitverse-code

该数据集是从GitVerse平台上的代码仓库中编译而来，GitVerse是一个俄罗斯的GitHub替代平台，预计在2024年初对所有用户开放。数据集包含了来自9,014个仓库的代码，涵盖了419种不同的文件类型，并经过去重和过滤处理，移除了二进制文件，最终从超过400 GB的分析数据中提取了30 GB的唯一代码。数据集包含2,804,216个唯一的代码文件，每个条目代表一个文件，包括其内容、识别的语言和唯一的文件名。数据集支持多种编程语言，每种文件的编程语言均通过github-linguist进行识别。数据集的结构包括文件内容、编程语言和文件名等字段，所有示例均位于训练集，没有验证集。数据集的创建者强调了伦理考虑，确保数据的收集和使用符合伦理规范，用户应尊重作者权利并负责任地使用数据。

The GitVerse Code Dataset is compiled from code repositories hosted on the GitVerse platform, a Russian alternative to GitHub. The dataset includes code from 9,014 repositories, covering 419 different file types identified by github-linguist. It consists of 30 GB of unique code extracted from over 400 GB of analyzed data, with 2,804,216 unique code files processed. Each entry in the dataset represents a single file, containing its content, identified language, and a unique filename. The dataset is multilingual and includes multiple programming languages, with each files language identified using github-linguist. The dataset fields include file_text, language, and file_name. All examples are in the train split, with no validation split. The dataset has been curated ethically, ensuring the collection and use of data respect the rights of the authors.