haggis

Overview of Data

This dataset cosists of XML containing stack overflow questions and answers as of August 2012 that can be used for text mining. The stack overflow dataset contains code snippets used by Haggis, a system for mining code idioms developed by the researchers.

The GitHub Java Corpus is a collection of Java code at a large scale, while it has been filtered to have an above-average quality (see next section). It can be used to study coding practice in Java at a large scale. It contains all project code, tracked by git with all the associated files. The containing open-source projects are from multiple domains. The code is not ours but is open-source. Please respect the license of each project.

How the corpus was collected

We processed GitHub's event stream, provided by the [GitHub Archive](http://www.githubarchive.org/) and filtered individual projects that were forked at least once. For each of these projects we collected the URL, the project's name and its language. Filtering by forks aims to create a quality corpus: Since we filter away the majority of projects that have a below-average "reputation", we obtain code of better quality. This criterion is not clear-cut and we could have chosen another method (e.g. GitHub's star system), but we found this preprocessing sufficient.

We then downloaded (clone in Git's terms) the full repositories for all remaining Java projects and removed any duplicates: We found about 1,000 projects that shared common commit SHAs indicating that they were most likely forks of the same project (but not declared on GitHub) and we manually picked those projects that seemed to be the original source. Project duplicates need to be removed to avoid a "leak" of data from the test to the train projects and also to create a more representative corpus of repositories.

We split the repository 75%-25% based on the lines of code assigning projects into either the training or test set. The split can be found in the RepositoryState.tar.gz

Top projects in the corpus

We have also ordered the projects in the corpus according to popularity, defined as the number of forks plus the number of watchers, where each is first seprately normalized into a z-score. The ordered list can be downloaded as _top\_projects.tar.gz_.

Corpus Statistics

Statistic                    | Train              | Test
---------------------------|--------------------|---------
Number of Projects  | 10,968           | 3,817
LOC                         |264,225,189   | 88,087,507
Tokens                     |1,116,195,158| 385,419,678