shailja/Verilog_GitHub

--- license: mit --- --- pipeline_tag: text-generation tags: - code model-index: - name: VeriGen results: - task: type: text-generation dataset: type: name: extra_gated_prompt: >- ## Model License Agreement Please read the BigCode [OpenRAIL-M license](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) agreement before accepting it. extra_gated_fields: I accept the above license agreement, and will use the Model complying with the set of use restrictions and sharing requirements: checkbox --- # VeriGen ## Table of Contents 1. [Dataset Summary](##model-summary) 2. [Use](##use) 3. [Limitations](##limitations) 4. [License](##license) 5. [Citation](##citation) ## Dataset Summary - The dataset comprises Verilog modules as entries. The entries were retrieved from the GitHub dataset on BigQuery. - For training [models (https://huggingface.co/shailja/fine-tuned-codegen-2B-Verilog)], we filtered entries with no of characters exceeding 20000 and duplicates (exact duplicates ignoring whitespaces). - **Paper:** [ Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://arxiv.org/abs/2212.11140) - **Point of Contact:** [contact@shailja](mailto:[e-mail]) - **Languages:** Verilog (Hardware Description Language) ### Data Splits The dataset only contains a train split. ### Use ```python # pip install datasets from datasets import load_dataset ds = load_dataset("shailja/Verilog_GitHub", streaming=True, split="train") print(next(iter(ds))) #OUTPUT: ``` ### Intended Use The dataset consists of source code from a range of GitHub repositories. As such, they can potentially include non-compilable, low-quality, and vulnerable code. ### Attribution & Other Requirements The pretraining dataset of the model was not filtered for permissive licenses only. Nevertheless, the model can generate source code verbatim from the dataset. The code's license might require attribution and/or other specific requirements that must be respected. # License The dataset is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement [here](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement). # Citation ``` @misc{https://doi.org/10.48550/arxiv.2212.11140, doi = {10.48550/ARXIV.2212.11140}, url = {https://arxiv.org/abs/2212.11140}, author = {Thakur, Shailja and Ahmad, Baleegh and Fan, Zhenxing and Pearce, Hammond and Tan, Benjamin and Karri, Ramesh and Dolan-Gavitt, Brendan and Garg, Siddharth}, title = {Benchmarking Large Language Models for Automated Verilog RTL Code Generation}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license} } ```许可证：MIT --- --- 流水线任务类型：文本生成 标签： - 代码 模型索引： - 名称：VeriGen 结果： - 任务： 类型：文本生成 数据集： 类型： 名称： 额外申请访问提示：> ## 模型许可协议 请在接受前阅读BigCode的[OpenRAIL-M许可协议](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement)。 额外申请访问字段： 我同意上述许可协议，并将遵守本模型的使用限制与共享要求：复选框 --- # VeriGen ## 目录 1. [数据集概述](##model-summary) 2. [使用方法](##use) 3. [局限性](##limitations) 4. [许可证](##license) 5. [引用](##citation) ## 数据集概述 - 本数据集以Verilog模块作为条目，所有条目源自BigQuery上的GitHub数据集。 - 针对训练[模型(https://huggingface.co/shailja/fine-tuned-codegen-2B-Verilog)]，我们过滤掉了字符数超过20000的条目以及重复项（忽略空白符的精确重复内容）。 - **论文**：[面向自动化Verilog寄存器传输级（RTL）代码生成的大语言模型（Large Language Model，LLM）基准测试](https://arxiv.org/abs/2212.11140) - **联络人**：[contact@shailja](mailto:[e-mail]) - **所用语言**：Verilog（硬件描述语言） ### 数据划分 本数据集仅包含训练划分。 ### 使用方法 python # 安装datasets库 # pip install datasets from datasets import load_dataset ds = load_dataset("shailja/Verilog_GitHub", streaming=True, split="train") print(next(iter(ds))) # 输出： ### 预期用途 本数据集包含来自各类GitHub仓库的源代码。因此，其中可能包含无法编译、质量低下或存在安全漏洞的代码。 ### 归属与其他要求 本模型的预训练数据集并非仅筛选了采用允许式许可证的代码。尽管如此，模型仍可能从数据集中逐字生成源代码。此类代码的许可证可能要求署名及/或其他需遵守的特定要求。 # 许可证 本数据集遵循BigCode OpenRAIL-M v1许可协议。完整协议可参阅[此处](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement)。 # 引用 @misc{https://doi.org/10.48550/arxiv.2212.11140, doi = {10.48550/ARXIV.2212.11140}, url = {https://arxiv.org/abs/2212.11140}, author = {Thakur, Shailja and Ahmad, Baleegh and Fan, Zhenxing and Pearce, Hammond and Tan, Benjamin and Karri, Ramesh and Dolan-Gavitt, Brendan and Garg, Siddharth}, title = {Benchmarking Large Language Models for Automated Verilog RTL Code Generation}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license} }

原始信息汇总

数据集概述

数据集内容

• 类型: Verilog模块
• 来源: 从GitHub数据集在BigQuery中检索
• 筛选条件: 排除字符数超过20000的条目及重复条目（忽略空格的完全重复）

数据集用途

• 训练模型: 用于训练特定的Verilog语言模型
• 数据集结构: 仅包含训练集

数据集特点

• 语言: Verilog（硬件描述语言）
• 潜在问题: 可能包含非编译、低质量及有漏洞的代码

许可证

• 类型: BigCode OpenRAIL-M v1许可证
• 要求: 可能需要对生成的源代码进行归属和遵守特定要求

引用信息

@misc{https://doi.org/10.48550/arxiv.2212.11140, doi = {10.48550/ARXIV.2212.11140}, url = {https://arxiv.org/abs/2212.11140}, author = {Thakur, Shailja and Ahmad, Baleegh and Fan, Zhenxing and Pearce, Hammond and Tan, Benjamin and Karri, Ramesh and Dolan-Gavitt, Brendan and Garg, Siddharth}, title = {Benchmarking Large Language Models for Automated Verilog RTL Code Generation}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license} }

搜集汇总

背景与挑战

背景概述

该数据集是一个 Verilog 代码集合，包含约 109k 行代码，来源于 GitHub 仓库，专门用于训练和评估大型语言模型在 Verilog RTL 代码生成任务中的性能。数据集经过过滤处理，去除了长字符条目和重复代码，并基于相关研究论文构建，适用于硬件描述语言的自动代码生成研究与应用。