NarsAI/google-code-archive

---
annotations_creators:
- machine-generated
language_creators:
- found
language:
- code
- en
license: other
multilinguality:
- multilingual
pretty_name: Google Code Archive Dataset
size_categories:
- 10M<n<100M
source_datasets:
- original
task_categories:
- text-generation
tags:
- code
- google-code
- archive
configs:
- config_name: default
data_files:
- split: train
path: "data/*.parquet"
default: true
dataset_info:
features:
- name: code
dtype: string
- name: repo_name
dtype: string
- name: path
dtype: string
- name: language
dtype: string
- name: license
dtype: string
- name: size
dtype: int64
---

# Google Code Archive Dataset

## Dataset Description

This dataset was compiled from the [Google Code Archive](https://code.google.com/archive/), a preserved snapshot of projects hosted on Google Code, Google's open-source project hosting service that operated from 2006 to 2016. Google Code was one of the major code hosting platforms of its era, hosting hundreds of thousands of open-source projects before its shutdown. The archive provides a unique historical record of open-source development during a formative period of modern software engineering.

### Dataset Summary

| Statistic | Value |
|-----------|-------|
| **Total Files** | 65,825,565 |
| **Total Repositories** | 488,618 |
| **Total Size** | 47 GB (compressed Parquet) |
| **Programming Languages** | 454 |
| **File Format** | Parquet with Zstd compression (71 files) |

### Key Features

- **Historical open-source corpus**: Contains code from over 488K repositories hosted on Google Code during 2006-2016
- **Diverse language coverage**: Spans 454 programming languages identified by [go-enry](https://github.com/go-enry/go-enry) (based on GitHub Linguist rules)
- **Rich metadata**: Includes repository name, file path, detected language, license information, and file size
- **Quality filtered**: Extensive filtering to remove vendor code, build artifacts, generated files, and low-quality content
- **Era-specific patterns**: Captures coding conventions and library usage from the pre-modern era of software development

### Languages

The dataset includes 454 programming languages. The top 30 languages by file count:

| Rank | Language | File Count |
|------|----------|------------|
| 1 | Java | 16,331,993 |
| 2 | PHP | 12,764,574 |
| 3 | HTML | 5,705,184 |
| 4 | C++ | 5,090,685 |
| 5 | JavaScript | 4,937,765 |
| 6 | C | 4,179,202 |
| 7 | C# | 3,872,245 |
| 8 | Python | 2,207,240 |
| 9 | CSS | 1,697,385 |
| 10 | Objective-C | 1,186,050 |
| 11 | Shell | 639,183 |
| 12 | Java Server Pages | 541,498 |
| 13 | ActionScript | 540,557 |
| 14 | Makefile | 481,563 |
| 15 | ASP.NET | 381,389 |
| 16 | Smarty | 339,555 |
| 17 | Ruby | 331,743 |
| 18 | Go | 316,427 |
| 19 | Perl | 307,960 |
| 20 | Vim Script | 216,236 |
| 21 | Lua | 215,226 |
| 22 | HTML+PHP | 150,781 |
| 23 | HTML+Razor | 149,131 |
| 24 | MATLAB | 145,686 |
| 25 | Batchfile | 138,523 |
| 26 | Pascal | 135,992 |
| 27 | Visual Basic .NET | 118,732 |
| 28 | TeX | 110,379 |
| 29 | Less | 98,221 |
| 30 | Unix Assembly | 94,758 |

### Licenses

The dataset includes files from repositories with various licenses as specified in the Google Code Archive:

| License | File Count |
|---------|------------|
| Apache License 2.0 (asf20) | 21,568,143 |
| GNU GPL v3 (gpl3) | 14,843,470 |
| GNU GPL v2 (gpl2) | 6,824,185 |
| Other Open Source (oos) | 5,433,436 |
| MIT License (mit) | 4,754,567 |
| GNU LGPL (lgpl) | 4,073,137 |
| BSD License (bsd) | 3,787,348 |
| Artistic License (art) | 1,910,047 |
| Eclipse Public License (epl) | 1,587,289 |
| Mozilla Public License 1.1 (mpl11) | 580,102 |
| Multiple Licenses (multiple) | 372,457 |
| Google Summer of Code (gsoc) | 63,292 |
| Public Domain (publicdomain) | 28,092 |

## Dataset Structure

### Data Fields

| Field | Type | Description |
|-------|------|-------------|
| `code` | string | Content of the source file (UTF-8 encoded) |
| `repo_name` | string | Name of the Google Code project |
| `path` | string | Path of the file within the repository (relative to repo root) |
| `language` | string | Programming language as identified by [go-enry](https://github.com/go-enry/go-enry) |
| `license` | string | License of the repository (Google Code license identifier) |
| `size` | int64 | Size of the source file in bytes |

### Data Format

- **Format**: Apache Parquet with Zstd compression
- **File Structure**: 71 files (`google_code_0000.parquet` to `google_code_0070.parquet`)

### Data Splits

All examples are in the train split. There is no validation or test split.

### Example Data Point

```
{
'code': 'public class HundredIntegers {\n\tpublic static void main (String[] args) {\n\t\tfor (int i = 1; i<=100; i++) {\n\t\t\tSystem.out.println(i);\n\t\t}\n\t}\n}',
'repo_name': '100integers',
'path': 'HundredIntegers.java',
'language': 'Java',
'license': 'epl',
'size': 147
}
```

## Dataset Creation

### Project Discovery

Most project IDs were sourced from the [Archive Team Google Code Wiki](https://wiki.archiveteam.org/index.php/Google_Code). This list provided the basis for the crawling and extraction process.

### Pipeline Overview

The dataset was created through a multi-stage pipeline:

1. **Project Discovery**: Fetching project metadata from the Google Code Archive
2. **Source Filtering**: Selecting projects that have source code available (`hasSource: true`)
3. **Archive Downloading**: Downloading source archives from the Google Code Archive storage
4. **Content Extraction**: Extracting and filtering source code files
5. **Parquet Generation**: Writing filtered records to Parquet shards with Zstd compression

### Language Detection

Programming languages are detected using [go-enry](https://github.com/go-enry/go-enry), a Go port of GitHub's Linguist library. Only files classified as **Programming** or **Markup** language types are included (Data and Prose types are excluded).

### License Detection

Licenses are obtained directly from the Google Code Archive project metadata. The archive preserves the original license selection made by project owners when creating their repositories on Google Code.

### File Filtering

Extensive filtering is applied to ensure data quality:

#### Size Limits
| Limit | Value |
|-------|-------|
| Max repository archive size | 64 MB |
| Max single file size | 2 MB |
| Max line length | 1,000 characters |

#### Excluded Directories
- **Configuration**: `.git/`, `.github/`, `.gitlab/`, `.vscode/`, `.idea/`, `.vs/`, `.settings/`, `.eclipse/`, `.project/`, `.metadata/`
- **Vendor/Dependencies**: `node_modules/`, `bower_components/`, `jspm_packages/`, `vendor/`, `third_party/`, `3rdparty/`, `external/`, `packages/`, `deps/`, `lib/vendor/`, `target/dependency/`, `Pods/`
- **Build Output**: `build/`, `dist/`, `out/`, `bin/`, `target/`, `release/`, `debug/`, `.next/`, `.nuxt/`, `_site/`, `_build/`, `__pycache__/`, `.pytest_cache/`, `cmake-build-*`, `.gradle/`, `.maven/`

#### Excluded Files
- **Lock Files**: `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `Gemfile.lock`, `Cargo.lock`, `poetry.lock`, `Pipfile.lock`, `composer.lock`, `go.sum`, `mix.lock`
- **Minified Files**: Any file containing `.min.` in the name
- **Binary Files**: `.exe`, `.dll`, `.so`, `.dylib`, `.a`, `.lib`, `.o`, `.obj`, `.jar`, `.war`, `.ear`, `.class`, `.pyc`, `.pyo`, `.wasm`, `.bin`, `.dat`, `.pdf`, `.doc`, `.docx`, `.xls`, `.xlsx`, `.ppt`, `.pptx`, `.zip`, `.tar`, `.gz`, `.bz2`, `.7z`, `.rar`, `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.ico`, `.svg`, `.mp3`, `.mp4`, `.avi`, `.mov`, `.wav`, `.flac`, `.ttf`, `.otf`, `.woff`, `.woff2`, `.eot`
- **System Files**: `.DS_Store`, `thumbs.db`

#### Content Filtering
- **UTF-8 Validation**: Files must be valid UTF-8 encoded text
- **Binary Detection**: Files detected as binary by go-enry are excluded
- **Generated Files**: Files with generation markers in the first 500 bytes are excluded:
- `generated by`, `do not edit`, `auto-generated`, `autogenerated`, `@generated`, `<auto-generated`
- **Empty Files**: Files that are empty or contain only whitespace are excluded
- **Long Lines**: Files with any line exceeding 1,000 characters are excluded
- **go-enry Filters**: Additional filtering using go-enry's `IsVendor()`, `IsImage()`, `IsDotFile()`, `IsTest()`, and `IsGenerated()` functions
- **Documentation-only Repos**: Repositories containing only documentation files (no actual code) are skipped

### Source Data

All data originates from the [Google Code Archive](https://code.google.com/archive/), which preserves projects hosted on Google Code before its shutdown in January 2016.

## Considerations for Using the Data

### Historical Context

This dataset represents code from 2006-2016 and may contain:
- Outdated coding patterns and deprecated APIs
- Legacy library dependencies that are no longer maintained
- Security vulnerabilities that have since been discovered and patched
- Code written for older language versions (Python 2, older Java versions, etc.)

Users should be aware that this code reflects historical practices and may not represent modern best practices.

### Personal and Sensitive Information

The dataset may contain:
- Email addresses in code comments or configuration files
- API keys or credentials that were accidentally committed
- Personal information in comments or documentation

Users should exercise caution and implement appropriate filtering when using this data.

### Licensing Information

This dataset is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in this dataset must abide by the terms of the original licenses, including attribution clauses when relevant. The license field in each data point indicates the license of the source repository.

---
注释创建者:
- 机器生成
语言创建方式:
- 现有资源采集
语言:
- 代码
- 英语
许可证: 其他
多语言属性: 多语言
展示名称: 谷歌代码档案数据集
规模分类:
- 1000万 < 样本数 < 1亿
源数据集:
- 原始数据集
任务分类:
- 文本生成
标签:
- 代码
- 谷歌代码
- 档案
配置项:
- 配置名称: 默认
数据文件:
- 拆分: 训练集
路径: "data/*.parquet"
默认配置: 是
数据集信息:
字段列表:
- 字段名: code
数据类型: 字符串
- 字段名: repo_name
数据类型: 字符串
- 字段名: path
数据类型: 字符串
- 字段名: language
数据类型: 字符串
- 字段名: license
数据类型: 字符串
- 字段名: size
数据类型: 64位整数
---

# 谷歌代码档案数据集

## 数据集说明

本数据集源自[谷歌代码档案库](https://code.google.com/archive/)，该档案库留存了2006年至2016年间谷歌开源项目托管服务Google Code上托管的所有项目快照。Google Code是当时主流的代码托管平台之一，在关停前承载了数十万开源项目。该档案库为现代软件工程形成期的开源开发历程提供了独一无二的历史记录。

## 数据集概览

| 统计项 | 数值 |
|-----------|-------|
| **总文件数** | 65,825,565 |
| **总仓库数** | 488,618 |
| **总规模** | 47 GB（压缩后的Parquet格式） |
| **支持编程语言数** | 454 |
| **文件格式** | 采用Zstd压缩的Parquet格式（共71个文件） |

## 核心特性

- **历史开源语料库**：收录了2006-2016年间Google Code平台上超过48.8万个托管仓库的代码
- **多语言覆盖**：涵盖454种编程语言，由[go-enry](https://github.com/go-enry/go-enry)（基于GitHub Linguist规则开发）识别
- **丰富元数据**：包含仓库名称、文件相对仓库根目录的路径、识别出的编程语言、许可证信息以及文件大小
- **质量过滤处理**：经过多轮严格过滤，移除了第三方依赖代码、构建产物、自动生成文件以及低质量内容
- **时代特征鲜明**：留存了现代软件开发早期的编码规范与第三方库使用习惯

## 编程语言分布

本数据集涵盖454种编程语言，以下为按文件数量排序的前30种语言：

| 排名 | 编程语言 | 文件数量 |
|------|----------|------------|
| 1 | Java | 16,331,993 |
| 2 | PHP | 12,764,574 |
| 3 | HTML | 5,705,184 |
| 4 | C++ | 5,090,685 |
| 5 | JavaScript | 4,937,765 |
| 6 | C | 4,179,202 |
| 7 | C# | 3,872,245 |
| 8 | Python | 2,207,240 |
| 9 | CSS | 1,697,385 |
| 10 | Objective-C | 1,186,050 |
| 11 | Shell | 639,183 |
| 12 | Java Server Pages | 541,498 |
| 13 | ActionScript | 540,557 |
| 14 | Makefile | 481,563 |
| 15 | ASP.NET | 381,389 |
| 16 | Smarty | 339,555 |
| 17 | Ruby | 331,743 |
| 18 | Go | 316,427 |
| 19 | Perl | 307,960 |
| 20 | Vim Script | 216,236 |
| 21 | Lua | 215,226 |
| 22 | HTML+PHP | 150,781 |
| 23 | HTML+Razor | 149,131 |
| 24 | MATLAB | 145,686 |
| 25 | Batchfile | 138,523 |
| 26 | Pascal | 135,992 |
| 27 | Visual Basic .NET | 118,732 |
| 28 | TeX | 110,379 |
| 29 | Less | 98,221 |
| 30 | Unix Assembly | 94,758 |

## 许可证分布

本数据集包含Google Code档案库中各仓库对应的各类许可证文件：

| 许可证 | 文件数量 |
|---------|------------|
| Apache许可证2.0（asf20） | 21,568,143 |
| GNU GPL v3（gpl3） | 14,843,470 |
| GNU GPL v2（gpl2） | 6,824,185 |
| 其他开源协议（oos） | 5,433,436 |
| MIT许可证（mit） | 4,754,567 |
| GNU LGPL（lgpl） | 4,073,137 |
| BSD许可证（bsd） | 3,787,348 |
| Artistic许可证（art） | 1,910,047 |
| Eclipse公共许可证（epl） | 1,587,289 |
| Mozilla公共许可证1.1（mpl11） | 580,102 |
| 多许可证（multiple） | 372,457 |
| Google编程之夏（gsoc） | 63,292 |
| 公共领域（publicdomain） | 28,092 |

## 数据集结构

### 数据字段

| 字段名 | 数据类型 | 字段说明 |
|-------|------|-------------|
| `code` | 字符串 | 源代码文件内容（UTF-8编码） |
| `repo_name` | 字符串 | Google Code项目的名称 |
| `path` | 字符串 | 文件在仓库内的相对路径（相对于仓库根目录） |
| `language` | 字符串 | 由[go-enry](https://github.com/go-enry/go-enry)识别出的编程语言 |
| `license` | 字符串 | 仓库对应的许可证（使用Google Code的许可证标识符） |
| `size` | 64位整数 | 源代码文件的字节大小 |

### 数据格式

- **格式**：采用Zstd压缩的Apache Parquet格式
- **文件结构**：共71个文件（命名格式为`google_code_0000.parquet`至`google_code_0070.parquet`）

### 数据拆分

所有数据样本均归入训练集拆分，无验证集与测试集拆分。

### 数据样本示例

{
'code': 'public class HundredIntegers {
public static void main (String[] args) {
for (int i = 1; i<=100; i++) {
System.out.println(i);
}
}
}',
'repo_name': '100integers',
'path': 'HundredIntegers.java',
'language': 'Java',
'license': 'epl',
'size': 147
}

## 数据集构建流程

### 项目发现

绝大多数项目ID源自[Archive Team谷歌代码维基](https://wiki.archiveteam.org/index.php/Google_Code)，该列表为后续的爬取与提取流程提供了基础。

### 整体构建流程

本数据集通过多阶段流水线构建而成：
1. **项目发现**：从Google Code档案库拉取项目元数据
2. **源代码筛选**：筛选出标注了`hasSource: true`的拥有可用源代码的项目
3. **档案下载**：从Google Code档案库存储节点下载源代码归档文件
4. **内容提取**：提取并过滤源代码文件
5. **Parquet文件生成**：将过滤后的样本写入采用Zstd压缩的Parquet分片文件

### 语言识别

编程语言识别采用[go-enry](https://github.com/go-enry/go-enry)工具，该工具是GitHub Linguist库的Go语言移植版本。仅保留被归类为**编程语言**或**标记语言**的文件（排除数据类与文本类文件）。

### 许可证识别

许可证信息直接源自Google Code档案库的项目元数据，该档案库完整保留了项目所有者在Google Code平台创建仓库时选定的原始许可证。

### 文件过滤

为保障数据质量，本数据集采用了多维度的过滤规则：

#### 大小限制

| 限制项 | 限制值 |
|-------|-------|
| 最大仓库归档文件大小 | 64 MB |
| 最大单文件大小 | 2 MB |
| 最大单行字符数 | 1000字符 |

#### 需排除的目录

- **配置目录**：`.git/`、`.github/`、`.gitlab/`、`.vscode/`、`.idea/`、`.vs/`、`.settings/`、`.eclipse/`、`.project/`、`.metadata/`
- **第三方依赖目录**：`node_modules/`、`bower_components/`、`jspm_packages/`、`vendor/`、`third_party/`、`3rdparty/`、`external/`、`packages/`、`deps/`、`lib/vendor/`、`target/dependency/`、`Pods/`
- **构建产物目录**：`build/`、`dist/`、`out/`、`bin/`、`target/`、`release/`、`debug/`、`.next/`、`.nuxt/`、`_site/`、`_build/`、`__pycache__/`、`.pytest_cache/`、`cmake-build-*`、`.gradle/`、`.maven/`

#### 需排除的文件

- **锁文件**：`package-lock.json`、`yarn.lock`、`pnpm-lock.yaml`、`Gemfile.lock`、`Cargo.lock`、`poetry.lock`、`Pipfile.lock`、`composer.lock`、`go.sum`、`mix.lock`
- **压缩后文件**：文件名中包含`.min.`的文件
- **二进制文件**：`.exe`、`.dll`、`.so`、`.dylib`、`.a`、`.lib`、`.o`、`.obj`、`.jar`、`.war`、`.ear`、`.class`、`.pyc`、`.pyo`、`.wasm`、`.bin`、`.dat`、`.pdf`、`.doc`、`.docx`、`.xls`、`.xlsx`、`.ppt`、`.pptx`、`.zip`、`.tar`、`.gz`、`.bz2`、`.7z`、`.rar`、`.jpg`、`.jpeg`、`.png`、`.gif`、`.bmp`、`.ico`、`.svg`、`.mp3`、`.mp4`、`.avi`、`.mov`、`.wav`、`.flac`、`.ttf`、`.otf`、`.woff`、`.woff2`、`.eot`
- **系统文件**：`.DS_Store`、`thumbs.db`

#### 内容过滤

- **UTF-8合法性校验**：文件必须为合法的UTF-8编码文本
- **二进制文件检测**：被go-enry识别为二进制的文件将被排除
- **自动生成文件过滤**：若文件前500字节中包含以下生成标记，则将其排除：`generated by`、`do not edit`、`auto-generated`、`autogenerated`、`@generated`、`<auto-generated`
- **空文件过滤**：空文件或仅包含空白字符的文件将被排除
- **超长行过滤**：包含任意长度超过1000字符的行的文件将被排除
- **go-enry附加过滤**：使用go-enry的`IsVendor()`、`IsImage()`、`IsDotFile()`、`IsTest()`以及`IsGenerated()`函数进行额外过滤
- **纯文档仓库过滤**：仅包含文档文件而无实际代码的仓库将被跳过

## 源数据来源

所有数据均源自[谷歌代码档案库](https://code.google.com/archive/)，该档案库完整留存了2016年1月关停前Google Code平台托管的所有项目。

## 数据集使用注意事项

### 历史局限性说明

本数据集收录的代码均为2006年至2016年间的产物，可能包含以下内容：
- 过时的编码范式与已废弃的API
- 不再维护的老旧第三方库依赖
- 后续已被发现并修复的安全漏洞
- 针对老旧语言版本编写的代码（如Python 2、旧版Java等）

用户需注意，本数据集的代码反映了当时的开发实践，未必符合现代软件工程的最佳实践。

### 个人与敏感信息提示

本数据集可能包含以下内容：
- 代码注释或配置文件中出现的电子邮箱地址
- 意外提交的API密钥或凭证信息
- 注释或文档中包含的个人信息

用户在使用该数据集时应保持谨慎，并采取适当的过滤措施。

### 许可证使用说明

本数据集收录了各类许可证对应的开源源代码，对本数据集全部或部分代码的任何使用都必须遵守原始许可证的条款，涉及署名要求时需遵循相关规定。每条数据样本中的`license`字段均标注了其所属仓库的许可证类型。