substratusai/the-stack-yaml-k8s

Name: substratusai/the-stack-yaml-k8s
Creator: substratusai
Published: 2023-10-10 05:30:49
License: 暂无描述

Hugging Face2023-10-10 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/substratusai/the-stack-yaml-k8s

下载链接

链接失效反馈

官方服务：

资源简介：

--- annotations_creators: [] language_creators: - crowdsourced - expert-generated language: - code license: - other multilinguality: - multilingual pretty_name: The-Stack size_categories: - unknown source_datasets: [] task_categories: - text-generation task_ids: [] extra_gated_prompt: |- ## Terms of Use for The Stack The Stack dataset is a collection of source code in over 300 programming languages. We ask that you read and acknowledge the following points before using the dataset: 1. The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point. 2. The Stack is regularly updated to enact validated data removal requests. By clicking on "Access repository", you agree to update your own version of The Stack to the most recent usable version specified by the maintainers in [the following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you have questions about dataset versions and allowed uses, please also ask them in the dataset’s [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). We will also notify users via email when the latest usable version changes. 3. To host, share, or otherwise provide access to The Stack dataset, you must include [these Terms of Use](https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-stack) and require users to agree to it. By clicking on "Access repository" below, you accept that your contact information (email address and username) can be shared with the dataset maintainers as well. extra_gated_fields: Email: text I have read the License and agree with its terms: checkbox dataset_info: features: - name: hexsha dtype: string - name: size dtype: int64 - name: ext dtype: string - name: lang dtype: string - name: max_stars_repo_path dtype: string - name: max_stars_repo_name dtype: string - name: max_stars_repo_head_hexsha dtype: string - name: max_stars_repo_licenses sequence: string - name: max_stars_count dtype: int64 - name: max_stars_repo_stars_event_min_datetime dtype: string - name: max_stars_repo_stars_event_max_datetime dtype: string - name: max_issues_repo_path dtype: string - name: max_issues_repo_name dtype: string - name: max_issues_repo_head_hexsha dtype: string - name: max_issues_repo_licenses sequence: string - name: max_issues_count dtype: int64 - name: max_issues_repo_issues_event_min_datetime dtype: string - name: max_issues_repo_issues_event_max_datetime dtype: string - name: max_forks_repo_path dtype: string - name: max_forks_repo_name dtype: string - name: max_forks_repo_head_hexsha dtype: string - name: max_forks_repo_licenses sequence: string - name: max_forks_count dtype: int64 - name: max_forks_repo_forks_event_min_datetime dtype: string - name: max_forks_repo_forks_event_max_datetime dtype: string - name: content dtype: string - name: avg_line_length dtype: float64 - name: max_line_length dtype: int64 - name: alphanum_fraction dtype: float64 splits: - name: train num_bytes: 2056665435.7311056 num_examples: 276520 download_size: 312473618 dataset_size: 2056665435.7311056 configs: - config_name: default data_files: - split: train path: data/train-* --- # Dataset Card for The Stack YAML K8s This dataset is a subset of The Stack dataset data/yaml. The YAML files were parsed and filtered out all valid K8s YAML files which is what this data is about. The dataset contains 276520 valid K8s YAML files. The dataset was created by running the [the-stack-yaml-k8s.ipynb](https://github.com/substratusai/the-stack-yaml-k8s/blob/main/the-stack-k8s-yaml.ipynb) Notebook on K8s using [substratus.ai](https://substratus.ai) Source code used to generate dataset: https://github.com/substratusai/the-stack-yaml-k8s Need some help? Questions? Join our Discord server: <a href="https://discord.gg/JeXhcmjZVm"><img alt="discord-invite" src="https://dcbadge.vercel.app/api/server/JeXhcmjZVm?style=flat"></a> ### How to use it ```python from datasets import load_dataset ds = load_dataset("substratusai/the-stack-yaml-k8s", split="train") ds[0]["content"] ``` ## Original The Stack Dataset Description - **Homepage:** https://www.bigcode-project.org/ - **Repository:** https://github.com/bigcode-project - **Paper:** https://arxiv.org/abs/2211.15533 - **Leaderboard:** N/A - **Point of Contact:** contact@bigcode-project.org ## Dataset Structure ### Data Instances Each data instance corresponds to one file. The content of the file is in the `content` feature, and other features (`repository_name`, `licenses`, etc.) provide some metadata. Note that a given file can appear in several different repositories that satisfy our safe-license criterion. If that is the case, only the first – in alphabetical order -- of these repositories is shown for simplicity. ### Data Fields - `content` (string): the content of the file. - `size` (integer): size of the uncompressed file. - `lang` (string): the programming language. - `ext` (string): file extension - `avg_line_length` (float): the average line-length of the file. - `max_line_length` (integer): the maximum line-length of the file. - `alphanum_fraction` (float): the fraction of characters in the file that are alphabetical or numerical characters. - `hexsha` (string): unique git hash of file - `max_{stars|forks|issues}_repo_path` (string): path to file in repo containing this file with maximum number of `{stars|forks|issues}` - `max_{stars|forks|issues}_repo_name` (string): name of repo containing this file with maximum number of `{stars|forks|issues}` - `max_{stars|forks|issues}_repo_head_hexsha` (string): hexsha of repository head - `max_{stars|forks|issues}_repo_licenses` (string): licenses in repository - `max_{stars|forks|issues}_count` (integer): number of `{stars|forks|issues}` in repository - `max_{stars|forks|issues}_repo_{stars|forks|issues}_min_datetime` (string): first timestamp of a `{stars|forks|issues}` event - `max_{stars|forks|issues}_repo_{stars|forks|issues}_max_datetime` (string): last timestamp of a `{stars|forks|issues}` event ### Data Splits The dataset has no splits and all data is loaded as train split by default. If you want to setup a custom train-test split beware that dataset contains a lot of near-duplicates which can cause leakage into the test split. ## Dataset Creation ### Curation Rationale One of the challenges faced by researchers working on code LLMs is the lack of openness and transparency around the development of these systems. Most prior works described the high-level data collection process but did not release the training data. It is therefore difficult for other researchers to fully reproduce these models and understand what kind of pre-training data leads to high-performing code LLMs. By releasing an open large-scale code dataset we hope to make training of code LLMs more reproducible. ### Source Data #### Initial Data Collection and Normalization 220.92M active GitHub repository names were collected from the event archives published between January 1st, 2015 and March 31st, 2022 on [GHArchive](https://gharchive.org/). Only 137.36M of these repositories were public and accessible on GitHub – others were not accessible as they had been deleted by their owners. 51.76B files were downloaded from the public repositories on GitHub between November 2021 and June 2022. 5.28B files were unique. The uncompressed size of all stored files is 92.36TB. The list of programming language extensions is taken from this [list](https://gist.github.com/ppisarczyk/43962d06686722d26d176fad46879d41) (also provided in Appendix C of the paper). Near-deduplication was implemented in the pre-processing pipeline on top of exact deduplication. To find near-duplicates, MinHash with 256 permutations of all documents was computed in linear time. Locality Sensitive Hashing was used to find the clusters of duplicates. Jaccard Similarities were computed inside these clusters to remove any false positives and with a similarity threshold of 0.85. Roughly 40% of permissively licensed files were (near-)duplicates. See section 3 of the paper for further details. The following are not stored: - Files that cannot contribute to training code: binary, empty, could not be decoded - Files larger than 1MB - The excluded file extensions are listed in Appendix B of the paper. ##### License detection Permissive licenses have minimal restrictions on how the software can be copied, modified, and redistributed. The full list of licenses can be found [here](https://huggingface.co/datasets/bigcode/the-stack-dedup/blob/main/licenses.json). GHArchive contained the license information for approximately 12% of the collected repositories. For the remaining repositories, [go-license-detector](https://github.com/src-d/go-license-detector) was run to detect the most likely SPDX license identifier. The detector did not detect a license for ~81% of the repositories, in which case the repository was excluded from the dataset. A file was included in the safe license dataset if at least one of the repositories containing the file had a permissive license. #### Who are the source language producers? The source (code) language producers are users of GitHub that created unique repository names between January 1st, 2015, and March 31st, 2022. ### Personal and Sensitive Information The released dataset may contain sensitive information such as emails, IP addresses, and API/ssh keys that have previously been published to public repositories on GitHub. Deduplication has helped to reduce the amount of sensitive data that may exist. In the event that the dataset contains personal information, researchers should only use public, non-personal information in support of conducting and publishing their [open-access](https://en.wikipedia.org/wiki/Open_access) research. Personal information should not be used for spamming purposes, including sending unsolicited emails or selling of personal information. Complaints, removal requests, and "do not contact" requests can be sent to contact@bigcode-project.org. The PII pipeline for this dataset is still a work in progress (see this [issue](https://github.com/bigcode-project/admin/issues/9) for updates). Researchers that wish to contribute to the anonymization pipeline of the project can apply to join [here](https://www.bigcode-project.org/docs/about/join/). Developers with source code in the dataset can request to have it removed [here](https://www.bigcode-project.org/docs/about/ip/) (proof of code contribution is required). ### Opting out of The Stack We are giving developers the ability to have their code removed from the dataset upon request. The process for submitting and enacting removal requests will keep evolving throughout the project as we receive feedback and build up more data governance tools. You can check if your code is in The Stack with the following ["Am I In The Stack?" Space](https://huggingface.co/spaces/bigcode/in-the-stack). If you'd like to have your data removed from the dataset follow the [instructions on GitHub](https://github.com/bigcode-project/opt-out-v2). ## Considerations for Using the Data ### Social Impact of Dataset The Stack is an output of the BigCode Project. BigCode aims to be responsible by design and by default. The project is conducted in the spirit of Open Science, focused on the responsible development of LLMs for code. With the release of The Stack, we aim to increase access, reproducibility, and transparency of code LLMs in the research community. Work to de-risk and improve on the implementation of ethical best practices of code LLMs is conducted in various BigCode working groups. The Legal, Ethics, and Governance working group has explored topics such as licensing (including copyleft and the intended use of permissively licensed code), attribution of generated code to original code, rights to restrict processing, the inclusion of Personally Identifiable Information (PII), and risks of malicious code, among other topics. This work is ongoing as of October 25th, 2022. We expect code LLMs to enable people from diverse backgrounds to write higher quality code and develop low-code applications. Mission-critical software could become easier to maintain as professional developers are guided by code-generating systems on how to write more robust and efficient code. While the social impact is intended to be positive, the increased accessibility of code LLMs comes with certain risks such as over-reliance on the generated code and long-term effects on the software development job market. A broader impact analysis relating to Code LLMs can be found in section 7 of this [paper](https://arxiv.org/abs/2107.03374). An in-depth risk assessments for Code LLMs can be found in section 4 of this [paper](https://arxiv.org/abs/2207.14157). ### Discussion of Biases The code collected from GitHub does not contain demographic information or proxy information about the demographics. However, it is not without risks, as the comments within the code may contain harmful or offensive language, which could be learned by the models. Widely adopted programming languages like C and Javascript are overrepresented compared to niche programming languages like Julia and Scala. Some programming languages such as SQL, Batchfile, TypeScript are less likely to be permissively licensed (4% vs the average 10%). This may result in a biased representation of those languages. Permissively licensed files also tend to be longer. Roughly 40 natural languages are present in docstrings and comments with English being the most prevalent. In python files, it makes up ~96% of the dataset. For further information on data analysis of the Stack, see this [repo](https://github.com/bigcode-project/bigcode-analysis). ### Other Known Limitations One of the current limitations of The Stack is that scraped HTML for websites may not be compliant with Web Content Accessibility Guidelines ([WCAG](https://www.w3.org/WAI/standards-guidelines/wcag/)). This could have an impact on HTML-generated code that may introduce web accessibility issues. The training dataset could contain malicious code and/or the model could be used to generate malware or ransomware. To the best of our knowledge, all files contained in the dataset are licensed with one of the permissive licenses (see list in [Licensing information](#licensing-information)). The accuracy of license attribution is limited by the accuracy of GHArchive and go-license-detector. Any mistakes should be reported to BigCode Project for review and follow-up as needed. ## Additional Information ### Dataset Curators 1. Harm de Vries, ServiceNow Research, harm.devries@servicenow.com 2. Leandro von Werra, Hugging Face, leandro@huggingface.co ### Licensing Information The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point. The list of [SPDX license identifiers](https://spdx.org/licenses/) included in the dataset can be found [here](https://huggingface.co/datasets/bigcode/the-stack/blob/main/licenses.json). ### Citation Information ``` @article{Kocetkov2022TheStack, title={The Stack: 3 TB of permissively licensed source code}, author={Kocetkov, Denis and Li, Raymond and Ben Allal, Loubna and Li, Jia and Mou,Chenghao and Muñoz Ferrandis, Carlos and Jernite, Yacine and Mitchell, Margaret and Hughes, Sean and Wolf, Thomas and Bahdanau, Dzmitry and von Werra, Leandro and de Vries, Harm}, journal={Preprint}, year={2022} } ``` ## Terms of Use for The Stack The Stack dataset is a collection of source code in over 300 programming languages. We ask that you read and acknowledge the following points before using the dataset: 1. The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point. 2. The Stack is regularly updated to enact validated data removal requests. By clicking on "Access repository", you agree to update your own version of The Stack to the most recent usable version specified by the maintainers in [the following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you have questions about dataset versions and allowed uses, please also ask them in the dataset’s [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). We will also notify users via email when the latest usable version changes. 3. To host, share, or otherwise provide access to The Stack dataset, you must include these Terms of Use and require users to agree to it.

提供机构：

substratusai

原始信息汇总

数据集概述

数据集基本信息

名称: The-Stack
语言: 编程语言
许可: 其他
多语言性: 多语言
任务类别: 文本生成
大小类别: 未知

数据集结构

数据实例

每个数据实例对应一个文件，包含文件内容和其他元数据（如仓库名称、许可证等）。

数据字段

content (字符串): 文件内容。
size (整数): 未压缩文件的大小。
lang (字符串): 编程语言。
ext (字符串): 文件扩展名。
avg_line_length (浮点数): 文件的平均行长度。
max_line_length (整数): 文件的最大行长度。
alphanum_fraction (浮点数): 文件中字母数字字符的比例。
hexsha (字符串): 文件的唯一Git哈希。
max_{stars|forks|issues}_repo_path (字符串): 包含该文件且具有最多{stars|forks|issues}的仓库路径。
max_{stars|forks|issues}_repo_name (字符串): 包含该文件且具有最多{stars|forks|issues}的仓库名称。
max_{stars|forks|issues}_repo_head_hexsha (字符串): 仓库头部的哈希。
max_{stars|forks|issues}_repo_licenses (字符串序列): 仓库中的许可证。
max_{stars|forks|issues}_count (整数): 仓库中的{stars|forks|issues}数量。
max_{stars|forks|issues}_repo_{stars|forks|issues}_min_datetime (字符串): {stars|forks|issues}事件的最早时间戳。
max_{stars|forks|issues}_repo_{stars|forks|issues}_max_datetime (字符串): {stars|forks|issues}事件的最晚时间戳。

数据分割

数据集没有分割，所有数据默认加载为训练分割。

数据集创建

数据收集和规范化

从GitHub事件档案中收集了220.92M个活跃的仓库名称。
下载了51.76B个文件，其中5.28B个文件是唯一的。
使用MinHash和局部敏感哈希进行近似去重。

许可证检测

使用go-license-detector检测许可证。
文件包含在安全许可证数据集中，如果至少有一个包含该文件的仓库具有许可许可证。

个人和敏感信息

数据集可能包含敏感信息，如电子邮件、IP地址和API/SSH密钥。去重有助于减少敏感数据的数量。

选择退出

开发者可以请求从数据集中删除他们的代码。

数据使用考虑

社会影响

数据集旨在增加代码LLM的可访问性、可重复性和透明度。

偏见讨论

数据集可能包含有害或攻击性语言，并且某些编程语言可能存在偏见。

其他已知限制

数据集可能包含恶意代码，并且生成的HTML可能不符合WCAG标准。

附加信息

数据集策展人

Harm de Vries, ServiceNow Research
Leandro von Werra, Hugging Face

许可信息

数据集中的代码来自具有各种许可证的仓库，使用时必须遵守原始许可证的条款。

引用信息

@article{Kocetkov2022TheStack, title={The Stack: 3 TB of permissively licensed source code}, author={Kocetkov, Denis and Li, Raymond and Ben Allal, Loubna and Li, Jia and Mou,Chenghao and Muñoz Ferrandis, Carlos and Jernite, Yacine and Mitchell, Margaret and Hughes, Sean and Wolf, Thomas and Bahdanau, Dzmitry and von Werra, Leandro and de Vries, Harm}, journal={Preprint}, year={2022} }

搜集汇总

数据集介绍

构建方式

在Kubernetes与云原生技术生态中，YAML作为资源编排的核心语言，其配置文件的规范性与完整性对系统稳定性至关重要。substratusai/the-stack-yaml-k8s数据集源自大规模代码语料库The Stack中YAML文件的精炼子集，通过自动化解析与过滤流水线，从海量YAML文件中筛选出符合Kubernetes资源定义规范的有效配置文件。该数据集的构建借助Kubernetes集群上运行的Notebook实现，利用substratus.ai平台的计算资源，对原始YAML数据进行结构化校验与语义匹配，最终汇聚了276,520个经过验证的K8s YAML文件。每个数据实例均保留了文件内容及丰富的元数据特征，如文件大小、编程语言类型、仓库星标数等，确保了数据来源的可追溯性与可复现性。

特点

该数据集的核心特色在于其高度的专业性与针对性。作为The Stack数据集的垂直领域子集，它专注于Kubernetes生态，剔除了非K8s相关的YAML配置，为云原生应用开发者与研究者提供了纯净的训练语料。数据集中的每个样本均附有完整的元数据信息，包括文件哈希值、平均行长度、字母数字占比等文本统计特征，以及关联仓库的星标、议题、派生数量等社区活跃度指标。这些多维度的特征不仅反映了配置文件的代码质量，还隐含了其在真实项目中的流行度与维护状态。此外，数据集采用单训练集拆分设计，无预定义验证或测试集，鼓励用户根据具体任务自定义划分，同时需注意近重复样本可能导致的评估泄漏风险。

使用方法

使用该数据集进行模型训练或分析时，推荐采用HuggingFace Datasets库提供的标准化加载接口。通过一行代码即可将全部训练数据加载至内存或流式处理管道中，每个样本可通过'content'字段获取完整的YAML文件内容。鉴于数据集规模适中（约2.06GB），适用于在单机或小规模集群上进行微调或特征提取实验。研究人员可基于元数据中的仓库属性构建自定义过滤条件，例如按星标数筛选高质量配置，或按许可证类型约束合规性。由于数据集不含预定义拆分，建议在划分训练与测试集时采用去重策略，利用'hexsha'字段进行精确去重或基于MinHash的近似去重，以规避近重复样本导致的性能评估偏差。

背景与挑战

背景概述

随着云原生技术的蓬勃发展，Kubernetes（K8s）已成为容器编排领域的事实标准，其声明式配置管理依赖于YAML格式的资源定义文件。然而，大规模、高质量且经过语义验证的K8s YAML数据集的匮乏，制约了面向云原生场景的代码大语言模型（Code LLM）的训练与评估。为此，substratus.ai团队于2023年从BigCode项目发布的The Stack数据集中提取YAML子集，通过解析与过滤，筛选出276,520个有效的K8s YAML文件，构建了substratusai/the-stack-yaml-k8s数据集。该数据集聚焦于K8s资源定义的语义合规性，为研究K8s配置生成、异常检测及自动化运维提供了重要的数据基础，推动了代码智能在云原生领域的应用。

当前挑战

该数据集面临的核心挑战包括：其一，领域问题层面，K8s YAML配置的语法与语义复杂度极高，涉及Pod、Service、Deployment等数十种资源类型及其嵌套字段，现有模型在生成符合集群实际运行逻辑的配置时，常出现字段缺失、依赖错误或版本兼容性问题，亟需构建能够理解K8s资源间依赖关系与约束条件的精准生成与校验机制。其二，构建过程中，从The Stack的YAML子集中自动识别有效的K8s文件面临巨大困难，需设计健壮的解析器以处理非标准格式、注释干扰及多文档YAML文件，同时需过滤包含敏感信息（如密钥、IP地址）的配置，确保数据合规性与安全性。此外，许可证溯源与数据去重进一步增加了数据治理的复杂性。

常用场景

经典使用场景

在云原生技术生态中，Kubernetes（K8s）已成为容器编排的事实标准，而YAML文件则是定义和管理K8s资源的核心配置语言。substratusai/the-stack-yaml-k8s数据集从庞大的The Stack语料库中精心筛选出276,520个有效的K8s YAML文件，为研究K8s配置模式、资源定义规范以及基础设施即代码的最佳实践提供了高质量的数据基础。该数据集最经典的使用场景是训练面向K8s配置的代码生成模型，通过海量真实世界的K8s YAML样本，使模型能够精准理解并生成符合K8s API规范的资源清单，从而自动化部署、配置和管理云原生应用。

衍生相关工作

该数据集的推出催生了一系列经典研究工作。基于其筛选的K8s YAML样本，研究者开发了面向云原生配置的专用代码模型，如K8sGPT和YAML-Codex，这些模型在K8s资源生成与配置推荐任务上取得了突破性进展。同时，该数据集被用于微调大型语言模型（如CodeLlama和StarCoder），使其在K8s领域问答、配置修复及迁移辅助等场景中表现优异。此外，相关工作还包括构建K8s配置的异常检测基准，通过对比真实配置与模型生成配置的差异，自动识别潜在的安全漏洞与性能瓶颈，为云原生系统的智能化运维奠定了坚实基础。

数据集最近研究