ronantakizawa/github-top-code

---
license: mit
task_categories:
- text-generation
language:
- code
tags:
- code
- github
- source-code
- trending-developers
- software-engineering
size_categories:
- 1M<n<10M
---

# GitHub Top Developer Source Code

A curated dataset of 1.3M+ source code files from **GitHub's top ranked developers (2015-2025)**.

This dataset is based on the top ranked developers from this dataset: https://huggingface.co/datasets/ronantakizawa/github-top-developers

## Dataset Summary

- **1.3M+ source code files** from repositories across ~4,700 unique developers
- **80+ programming languages** included (Python, JavaScript, TypeScript, Rust, Go, C/C++, Java, and more)
- **Source code only** — config files (JSON, YAML, TOML, etc.) and documentation (Markdown, TXT) are excluded
- **Permissive licenses only** (MIT, Apache-2.0, BSD, ISC, etc.)
- **Rich metadata** per file: repo stars, description, primary language, developer company affiliation


## Schema

Each row represents a single source file:

| Column | Type | Description |
|--------|------|-------------|
| `file_path` | string | Path within the repo (e.g. `src/main.py`) |
| `file_language` | string | Language detected from file extension (e.g. `Python`, `JavaScript`) |
| `content` | string | Raw source code (UTF-8) |
| `repo_name` | string | Full repository name (`owner/repo`) |
| `repo_stars` | int64 | GitHub star count at time of collection |
| `repo_description` | string | Repository description |
| `repo_primary_language` | string | GitHub-detected primary language of the repository |
| `developer_username` | string | GitHub username |
| `developer_name` | string | Developer display name |
| `developer_company` | string | Company affiliation |

**Note on language columns:** `file_language` is determined per-file from the file extension (e.g. a `.py` file is always `Python`). `repo_primary_language` is GitHub's auto-detected primary language for the entire repository. These may differ — for example, a C header file (`.h` → `C/C++ Header`) in a repo that GitHub classifies as `Python`.

## Splits

| Split | Description |
|-------|-------------|
| `train` | ~90% of repos — for training |
| `test` | ~5% of repos — for evaluation |
| `validation` | ~5% of repos — for hyperparameter tuning |

Splits are assigned **by repository** (deterministic hash), so no repo appears in multiple splits. This prevents data leakage from files in the same project.

## Usage

```python
from datasets import load_dataset

# Load a specific split
train = load_dataset("ronantakizawa/github-top-code", split="train")
test = load_dataset("ronantakizawa/github-top-code", split="test")

# Filter by language
python_files = train.filter(lambda x: x["file_language"] == "Python")

# Filter by stars
popular = train.filter(lambda x: x["repo_stars"] > 1000)

# Get files from a specific developer
dev_files = train.filter(lambda x: x["developer_username"] == "torvalds")
```

许可证：MIT
任务类别：文本生成
语言：代码
标签：代码、GitHub、源代码、热门开发者、软件工程
规模类别：100万 < n < 1000万

# GitHub热门开发者源代码数据集
本数据集为精选数据集，包含130万余个源代码文件，数据来源于**2015年至2025年GitHub排名前列的开发者**。
本数据集基于以下数据集中的热门开发者榜单：https://huggingface.co/datasets/ronantakizawa/github-top-developers

## 数据集概览
- **130万余个源代码文件**，覆盖约4700名独立开发者的代码仓库
- 涵盖**80余种编程语言**，包括Python、JavaScript、TypeScript、Rust、Go、C/C++、Java等
- 仅保留纯源代码，已排除配置文件（如JSON、YAML、TOML等）与文档文件（如Markdown、TXT等）
- 仅包含宽松许可证的代码（如MIT、Apache-2.0、BSD、ISC等）
- 每个文件附带丰富元数据：代码仓库星标数、仓库描述、主编程语言、开发者所属公司信息


## 数据结构
每一行对应一个独立源代码文件：

| 列名 | 数据类型 | 描述 |
|--------|------|-------------|
| `file_path` | string | 代码仓库内的文件路径（例如`src/main.py`） |
| `file_language` | string | 通过文件扩展名检测得到的编程语言（例如`Python`、`JavaScript`） |
| `content` | string | 原始源代码（UTF-8编码） |
| `repo_name` | string | 完整代码仓库名称（格式为`所有者/仓库名`） |
| `repo_stars` | int64 | 数据采集时该仓库的GitHub星标数 |
| `repo_description` | string | 代码仓库的描述信息 |
| `repo_primary_language` | string | GitHub自动检测的该代码仓库主编程语言 |
| `developer_username` | string | 开发者的GitHub用户名 |
| `developer_name` | string | 开发者的公开显示名称 |
| `developer_company` | string | 开发者所属公司信息 |

**关于语言列的说明**：`file_language` 为通过单个文件的扩展名检测得到的语言（例如`.py`文件始终被标记为`Python`）；`repo_primary_language` 为GitHub自动检测的整个代码仓库的主编程语言，二者可能存在差异——例如在一个被GitHub归类为`Python`的仓库中，可能存在C语言头文件（`.h` → 标记为`C/C++ Header`）。

## 数据集划分
| 划分集 | 描述 |
|-------|-------------|
| `train` | 约90%的代码仓库，用于模型训练 |
| `test` | 约5%的代码仓库，用于模型评估 |
| `validation` | 约5%的代码仓库，用于超参数调优 |

划分集**以代码仓库为单位进行分配**（通过确定性哈希实现），因此单个仓库不会同时出现在多个划分集中，可避免同一项目内的文件导致的数据泄露。

## 使用示例
python
from datasets import load_dataset

# 加载指定划分集
train = load_dataset("ronantakizawa/github-top-code", split="train")
test = load_dataset("ronantakizawa/github-top-code", split="test")

# 按编程语言筛选
python_files = train.filter(lambda x: x["file_language"] == "Python")

# 按星标数筛选
popular = train.filter(lambda x: x["repo_stars"] > 1000)

# 获取指定开发者的代码文件
dev_files = train.filter(lambda x: x["developer_username"] == "torvalds")