Code and Comments Dataset

The code and comment data are a compilation of code blocks and their related comments. Doxygen successfully ran on 106,304 different GitHub projects. A total of 16,115,540 code-comment pairs were obtained by running Doxygen on C, C++, Java, and Python projects. The source code in these pairs can be of various granularities: classes, methods, functions, and variables. These data provide an association between source code and a description of that code. The data directory contains one directory for each project downloaded from GitHub. These project directories are named with the GraphQL ID from GitHub's GraphQL API. In each of these GraphQL-ID labeled directories, there is a license.txt, a url.txt, and a derivatives directory. The license.txt contains the license for the original project, the url.txt contains a link to the original project on GitHub, and the derivatives directory contains the output of running Doxygen on the project. The Doxygen output is a json file, structured as a dictionary with a "contents" field, where the value of that field is a list of lists containing 3 elements each. The following is a mock example of that structure: {"contents": [[path1, snippet1, comment1], [path2, snippet2, comment2], ...]}. The "path" is a filepath relative to the original project from which the code and comment were obtained. The "snippet" is the actual body of the source code. The "comment" is the corresponding comment. For convenience, there is also an initialize.py python script that iterates through all of the json files in the data directory and stores them in an SQLite database called "all_data.db".本数据集的代码与注释数据为代码块及其相关注释的汇编集。Doxygen已成功在106304个不同的GitHub项目上运行。通过针对C、C++、Java及Python项目执行Doxygen，共获取到16115540组代码-注释对。这些代码-注释对中的源代码可涵盖多种粒度：类、方法、函数及变量。本数据集提供了源代码与其描述文本之间的关联关系。数据目录下为每个从GitHub下载的项目单独创建了子目录，这些子目录以GitHub GraphQL API返回的GraphQL ID命名。在每个以GraphQL ID命名的子目录中，均包含一个license.txt文件、一个url.txt文件以及一个derivatives目录。其中license.txt文件存储原始项目的开源协议信息，url.txt文件存储该项目在GitHub上的原始链接，derivatives目录则存储对该项目执行Doxygen后得到的输出结果。Doxygen的输出为JSON格式文件，其结构为包含"contents"字段的字典，该字段的值为一个列表，列表中的每个元素均为包含3个元素的子列表。以下为该结构的模拟示例：{"contents": [[path1, snippet1, comment1], [path2, snippet2, comment2], ...]}。其中"path"为相对于获取代码与注释的原始项目的文件路径，"snippet"为源代码的实际片段，"comment"为对应的注释文本。为便于使用，本数据集还附带了initialize.py Python脚本，该脚本可遍历数据目录下的所有JSON文件，并将其存储至名为"all_data.db"的SQLite数据库中。