Fsoft-AIC/the-vault-function
收藏Hugging Face2024-10-15 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/Fsoft-AIC/the-vault-function
下载链接
链接失效反馈官方服务：
资源简介：
---
language:
- code
- en
multilinguality:
- multiprogramming languages
task_categories:
- text-generation
license: mit
dataset_info:
features:
- name: identifier
dtype: string
- name: return_type
dtype: string
- name: repo
dtype: string
- name: path
dtype: string
- name: language
dtype: string
- name: code
dtype: string
- name: code_tokens
dtype: string
- name: original_docstring
dtype: string
- name: comment
dtype: string
- name: docstring_tokens
dtype: string
- name: docstring
dtype: string
- name: original_string
dtype: string
pretty_name: The Vault Function
viewer: true
---
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks](#supported-tasks)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Statistics](#dataset-statistics)
- [Usage](#usage)
- [Additional Information](#additional-information)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)
## Dataset Description
- **Repository:** [FSoft-AI4Code/TheVault](https://github.com/FSoft-AI4Code/TheVault)
- **Paper:** [The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation](https://arxiv.org/abs/2305.06156)
- **Contact:** support.ailab@fpt.com
- **Website:** https://www.fpt-aicenter.com/ai-residency/
<p align="center">
<img src="https://raw.githubusercontent.com/FSoft-AI4Code/TheVault/main/assets/the-vault-4-logo-png.png" width="300px" alt="logo">
</p>
<div align="center">
# The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
</div>
## Dataset Summary
The Vault dataset is a comprehensive, large-scale, multilingual parallel dataset that features high-quality code-text pairs derived from The Stack, the largest permissively-licensed source code dataset.
We provide The Vault which contains code snippets from 10 popular programming languages such as Java, JavaScript, Python, Ruby, Rust, Golang, C#, C++, C, and PHP. This dataset provides multiple code-snippet levels, metadata, and 11 docstring styles for enhanced usability and versatility.
## Supported Tasks
The Vault can be used for pretraining LLMs or downstream code-text interaction tasks. A number of tasks related to code understanding and geneartion can be constructed using The Vault such as *code summarization*, *text-to-code generation* and *code search*.
## Languages
The natural language text (docstring) is in English.
10 programming languages are supported in The Vault: `Python`, `Java`, `JavaScript`, `PHP`, `C`, `C#`, `C++`, `Go`, `Ruby`, `Rust`
## Dataset Structure
### Data Instances
```
{
"hexsha": "5c47f0b4c173a8fd03e4e633d9b3dd8211e67ad0",
"repo": "neumanna94/beepboop",
"path": "js/scripts.js",
"license": [
"MIT"
],
"language": "JavaScript",
"identifier": "beepBoopSelector",
"return_type": "<not_specific>",
"original_string": "function beepBoopSelector(inputString, bbFunction){\n if(bbFunction==1){\n return beepBoop(inputString);\n } else if(bbFunction==2){\n return beepBoop2(inputString);\n } else if(bbFunction==3){\n return beepBoop3(inputString);\n } else {\n }\n}",
"original_docstring": "//Determines what beepBoop function to use",
"docstring": "Determines what beepBoop function to use",
"docstring_tokens": [
"Determines",
"what",
"beepBoop",
"function",
"to",
"use"
],
"code": "function beepBoopSelector(inputString, bbFunction){\n if(bbFunction==1){\n return beepBoop(inputString);\n } else if(bbFunction==2){\n return beepBoop2(inputString);\n } else if(bbFunction==3){\n return beepBoop3(inputString);\n } else {\n }\n}",
"code_tokens": [
"function",
"beepBoopSelector",
"(",
"inputString",
",",
"bbFunction",
")",
"{",
"if",
"(",
"bbFunction",
"==",
"1",
")",
"{",
"return",
"beepBoop",
"(",
"inputString",
")",
";",
"}",
"else",
"if",
"(",
"bbFunction",
"==",
"2",
")",
"{",
"return",
"beepBoop2",
"(",
"inputString",
")",
";",
"}",
"else",
"if",
"(",
"bbFunction",
"==",
"3",
")",
"{",
"return",
"beepBoop3",
"(",
"inputString",
")",
";",
"}",
"else",
"{",
"}",
"}"
],
"short_docstring": "Determines what beepBoop function to use",
"short_docstring_tokens": [
"Determines",
"what",
"beepBoop",
"function",
"to",
"use"
],
"comment": [],
"parameters": [
{
"param": "inputString",
"type": null
},
{
"param": "bbFunction",
"type": null
}
],
"docstring_params": {
"returns": [],
"raises": [],
"params": [
{
"identifier": "inputString",
"type": null,
"docstring": null,
"docstring_tokens": [],
"default": null,
"is_optional": null
},
{
"identifier": "bbFunction",
"type": null,
"docstring": null,
"docstring_tokens": [],
"default": null,
"is_optional": null
}
],
"outlier_params": [],
"others": []
}
}
```
### Data Fields
Data fields for function level:
- **hexsha** (string): the unique git hash of file
- **repo** (string): the owner/repo
- **path** (string): the full path to the original file
- **license** (list): licenses in the repo
- **language** (string): the programming language
- **identifier** (string): the function or method name
- **return_type** (string): the type returned by the function
- **original_string** (string): original version of function/class node
- **original_docstring** (string): the raw string before tokenization or parsing
- **code** (string): the part of the original that is code
- **code_tokens** (list): tokenized version of `code`
- **short_docstring** (string): short, brief summarization (first line of the docstring)
- **short_docstring_tokens** (list): tokenized version of `short_docstring
- **docstring** (string): the top-level comment or docstring (docstring version without param’s doc, return, exception fields, etc)
- **docstring_tokens** (list): tokenized version of docstring
- **comment** (list): list of comments (line) inside the function/class
- **parameters** (list): List of parameters and its type (type can be None)
- **docstring_params** (dict): Dictionary of the parsed information from docstring
See [here](https://github.com/FSoft-AI4Code/TheVault/blob/main/data/README.md) for more details and examples.
### Data Splits
In this repo, The Vault is divided into 5 subsets, where three training versions are split based on size of the full training set, and the remains are validation set and test set (approximate 20,000 samples in each). The statistic for languages in each split set is illustrated in the following section.
Before split, the dataset is deduplicated. There are 3 versions of training set that are small (5%), medium (20%) and large (100%).
## Dataset Statistics
- Compare to other benchmarks
| Dataset | #Language | #Code-text pair |
|:--------------------------|----------:|-----------------:|
| PyMT5 | 1 | ≈ 7,700,000 |
| CoDesc | 1 | 4,211,516 |
| CodeSearchNet | 6 | 2,326,976 |
| CodeSearchNet (CodeXGLUE) | 6 | 1,005,474 |
| Deepcom | 1 | 424,028 |
| CONCODE | 1 | 2,184,310 |
| Funcom | 1 | 2,149,121 |
| CodeT5 | 8 | 3,158,313 |
| **The Vault** | **10** | **34,098,775** |
- Statistic for split sets
| | train/small | train/medium | train/full | validation | test | total |
|:-----------|------------:|-------------:|-----------:|-----------:|-------:|--------------:|
|Python | 370,657 | 1,952,110 | 7,772,647 | 30,992 | 21,652 | 7,825,291 |
|Java | 351,213 | 1,612,366 | 6,629,193 | 22,677 | 15,552 | 6,667,422 |
|JavaScript | 82,931 | 404,729 | 1,640,416 | 22,044 | 21,108 | 1,683,568 |
|PHP | 236,638 | 1,155,476 | 4,656,371 | 21,375 | 19,010 | 4,696,756 |
|C | 105,978 | 381,207 | 1,639,319 | 27,525 | 19,122 | 1,685,966 |
|C# | 141,090 | 783,166 | 3,305,891 | 24,787 | 19,638 | 3,350,316 |
|C++ | 87,420 | 410,907 | 1,671,268 | 20,011 | 18,169 | 1,709,448 |
|Go | 267,535 | 1,319,547 | 5,109,020 | 19,102 | 25,314 | 5,153,436 |
|Ruby | 23,921 | 112,574 | 424,339 | 17,338 | 19,908 | 461,585 |
|Rust | 35,367 | 224,015 | 825,130 | 16,716 | 23,141 | 864,987 |
|TOTAL | 1,702,750 | 8,356,097 |33,673,594 |222,567 |202,614 |**34,098,775** |
## Usage
You can load The Vault dataset using datasets library: ```pip install datasets```
```python
from datasets import load_dataset
# Load full function level dataset (34M samples)
dataset = load_dataset("Fsoft-AIC/the-vault-function")
# Load function level train/validation/test set
dataset = load_dataset("Fsoft-AIC/the-vault-function", split_set=["train"])
# Load "small" (or "medium", "full") version of function level training set
dataset = load_dataset("Fsoft-AIC/the-vault-function", split_set=["train/small"])
# specific language (e.g. Python)
dataset = load_dataset("Fsoft-AIC/the-vault-function", split_set=["train"], languages=['python'])
# dataset streaming
data = load_dataset("Fsoft-AIC/the-vault-function", split_set= ["train"], streaming= True)
for sample in iter(data['train']):
print(sample)
```
A back up dataset can be downloaded in azure storage. See [Download The Vault from Azure blob storage](https://github.com/FSoft-AI4Code/TheVault#download-via-link).
## Additional information
### Licensing Information
MIT License
### Citation Information
```
@article{manh2023vault,
title={The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation},
author={Manh, Dung Nguyen and Hai, Nam Le and Dau, Anh TV and Nguyen, Anh Minh and Nghiem, Khanh and Guo, Jin and Bui, Nghi DQ},
journal={arXiv preprint arXiv:2305.06156},
year={2023}
}
```
### Contributions
This dataset is developed by [FSOFT AI4Code team](https://github.com/FSoft-AI4Code).
提供机构：
Fsoft-AIC原始信息汇总
数据集概述
数据集名称
- 名称: The Vault Function
- 别名: The Vault
数据集描述
- 概述: The Vault是一个综合性的多语言数据集，用于推进代码理解和生成。它包含了从The Stack数据集中提取的高质量代码-文本对。
- 特点: 包含10种流行编程语言的代码片段，提供多种代码片段级别、元数据和11种文档字符串样式。
支持的任务
- 任务类型: 代码理解与生成
- 具体任务: 代码摘要生成、文本到代码生成、代码搜索等
支持的语言
- 编程语言: Python, Java, JavaScript, PHP, C, C#, C++, Go, Ruby, Rust
- 自然语言: 英语
数据集结构
- 数据实例: 每个实例包含代码片段及其相关元数据和文档字符串。
- 数据字段:
- hexsha: 文件的唯一git哈希
- repo: 仓库所有者/名称
- path: 原始文件的完整路径
- license: 仓库的许可证列表
- language: 编程语言
- identifier: 函数或方法名
- return_type: 函数返回类型
- original_string: 函数/类的原始版本
- original_docstring: 文档字符串的原始字符串
- code: 代码部分
- code_tokens: 代码的标记化版本
- short_docstring: 简短的文档字符串摘要
- short_docstring_tokens: 简短文档字符串的标记化版本
- docstring: 顶级注释或文档字符串
- docstring_tokens: 文档字符串的标记化版本
- comment: 函数/类内的注释列表
- parameters: 参数及其类型的列表
- docstring_params: 从文档字符串解析的信息字典
数据集统计
- 总计样本数: 34,098,775
- 训练集版本: 小(5%), 中(20%), 全(100%)
- 验证集和测试集样本数: 约20,000样本/集
使用方法
- 加载数据集: 使用
datasets库加载数据集，支持加载全量数据集、特定分割或特定语言的数据集。
许可证信息
- 许可证: MIT License
引用信息
@article{manh2023vault, title={The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation}, author={Manh, Dung Nguyen and Hai, Nam Le and Dau, Anh TV and Nguyen, Anh Minh and Nghiem, Khanh and Guo, Jin and Bui, Nghi DQ}, journal={arXiv preprint arXiv:2305.06156}, year={2023} }
贡献者
- 开发团队: FSOFT AI4Code团队
搜集汇总
数据集介绍
构建方式
在代码智能研究领域，高质量、大规模且多语言的对齐数据是推动模型理解与生成能力的关键基石。The Vault 数据集正是基于这一需求，从 The Stack 这一最大规模的开源许可代码数据集中，通过精细化的数据清洗与筛选流程构建而成。具体而言，该数据集聚焦于提取函数级别的代码片段，并为其配对了对应的文档字符串（docstring），从而形成了海量的代码-文本并行语料。构建过程中，研究者对原始数据进行了去重处理，并按照10种主流编程语言（如 Python、Java、C++ 等）进行分类，同时解析出11种不同的文档字符串风格，以确保数据的多样性与实用性。最终，数据集被划分为小、中、大三个规模的训练子集以及验证集和测试集，总计包含超过3400万个高质量样本。
使用方法
利用 The Vault 数据集进行模型训练与评估极为便捷，得益于 Hugging Face 的 datasets 库，用户可通过简单的 API 调用实现数据加载。例如，使用 `load_dataset("Fsoft-AIC/the-vault-function")` 即可获取包含全部3400万样本的完整数据集。若需按特定规模或语言筛选，可通过 `split_set` 参数指定训练子集（如 `"train/small"` 或 `"train/full"`），并通过 `languages` 参数限定编程语言（如 `["python"]`）。对于大规模数据场景，数据集支持流式加载模式，能够有效降低内存占用。此外，研究者还可通过 Azure 存储链接下载备份数据，以便在本地环境中进行灵活处理。这种模块化与可扩展的接口设计，显著降低了数据使用的门槛。
背景与挑战
背景概述
在代码智能领域，代码理解与生成任务的进步高度依赖于大规模、高质量且多语言的并行数据集。由FPT软件AI中心（FSOFT AI4Code团队）于2023年创建的The Vault数据集，正是为应对这一需求而生。该数据集基于The Stack这一最大规模的开源许可代码库构建，包含了Java、Python、JavaScript等10种主流编程语言的超过3400万个高质量代码-文本对，覆盖函数级别的多种元数据与11种文档字符串风格。其核心研究问题在于如何通过丰富的多语言语料库推动代码摘要、文本生成代码及代码搜索等下游任务的发展。凭借前所未有的规模和语言多样性，The Vault已成为预训练大型语言模型及评估代码智能基准的重要资源，显著提升了相关研究的可复现性与泛化能力。
当前挑战
The Vault数据集所面临的挑战首先体现在领域问题的复杂性上：代码理解与生成任务要求模型不仅掌握语法结构，还需理解语义逻辑与跨语言模式，而现有模型在处理长尾语言（如Ruby、Rust）时性能显著下降。其次，构建过程中遭遇多重技术难题：从The Stack海量数据中提取高质量代码-文本对时，需精确解析不同编程语言的函数边界并过滤噪声文档字符串；同时，为维护数据集的完整性与可用性，团队需应对跨语言文档字符串风格的巨大差异（如Javadoc与Rustdoc），并确保去重后训练集、验证集与测试集的分布平衡。这些挑战共同构成了推动代码智能研究向更鲁棒、更通用方向演进的核心瓶颈。
常用场景
经典使用场景
The Vault Function数据集以其海量的多语言代码-文本配对数据而著称，涵盖Java、Python、JavaScript等10种主流编程语言，总计超过3400万个函数级样本。其最经典的使用场景聚焦于代码摘要生成任务，即从函数体自动生成自然语言描述。研究者可借助该数据集训练模型，让模型学习代码逻辑与语义注释之间的映射关系，从而实现对任意代码片段的精准概括。此外，该数据集亦被广泛用于文本到代码生成，即根据自然语言指令自动合成对应的函数实现，为智能编程助手提供了坚实的数据基础。
解决学术问题
在学术研究层面，The Vault Function有效解决了多语言代码理解与生成领域中数据匮乏与标注不一致的瓶颈问题。传统数据集如CodeSearchNet仅覆盖6种语言且规模有限，而The Vault通过统一抽取来自The Stack的许可代码，构建了规模更大、语言覆盖更广且包含11种文档注释风格的标准化语料。这使研究者得以系统性地探索跨语言代码表示学习、代码搜索排序以及函数级语义匹配等核心问题，显著推动了代码智能领域在泛化能力和鲁棒性上的理论突破。
实际应用
在实际应用中，The Vault Function所驱动的模型已深度赋能开发者工具链。基于该数据集训练的代码摘要模型可集成至集成开发环境，在程序员编写函数时实时生成注释建议，降低文档维护成本。文本到代码生成能力则被应用于低代码平台，允许非专业用户通过自然语言描述快速生成可执行代码片段。此外，代码搜索功能借助该数据集优化的语义嵌入，实现了对大型代码仓库中相关函数的高效检索，极大提升了代码复用与调试效率，成为现代软件工程中不可或缺的智能基础设施。
数据集最近研究
最新研究方向
在代码智能领域，大型语言模型对高质量、多语言代码-文本对数据的需求日益迫切。The Vault数据集应运而生，它从The Stack中提取了涵盖10种主流编程语言的超过3400万个函数级代码-文本对，并提供了11种文档字符串风格，为代码理解与生成研究注入了新动力。当前，该数据集的前沿研究方向聚焦于利用其大规模、多语言特性来预训练和微调代码大语言模型，以提升模型在代码摘要、文本到代码生成以及代码搜索等下游任务上的表现。特别是在跨语言代码迁移学习和低资源语言代码生成等热点事件中，The Vault的丰富元数据和细粒度结构使得模型能够更精准地捕捉代码语义与文档之间的映射关系，从而推动代码智能的泛化能力与实用性迈向新高度。
以上内容由遇见数据集搜集并总结生成
