nareshmodina/TeleSpec-Data

--- license: cc-by-nc-4.0 language: - en tags: - telecom - LLM - ETSI - 3GPP - standards - continual-pretraining task_categories: - text-generation size_categories: - 10M<n<100M configs: - config_name: default data_files: - split: train path: data/*/*.parquet - config_name: 3gpp-standard data_files: - split: train path: data/3gpp-standard/*.parquet - config_name: etsi-standard data_files: - split: train path: data/etsi-standard/*.parquet --- # TeleSpec-Data ## Dataset Summary TeleSpec-Data is a dataset of telecommunications standards documents from two major bodies: ETSI and 3GPP. It is designed for continual pretraining of language models on telecom domain knowledge. The dataset consists of two categories: - **3gpp-standard**: 15,054 3GPP technical specifications and reports covering Release 8 through Release 19 (updated to April 2025), sourced from [TSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/TSpec-LLM). - **etsi-standard**: 23,248 ETSI documents spanning 15 working groups (TS, TR, EN, ES, GS, GR, EG, ETR, ETS, GTS, and others), covering years 2000–2024, extracted from the PDF corpus in [NetSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/NetSpec-LLM). **Total: 38,302 documents.** ## Dataset Structure ### Data Fields - `id`: Unique identifier for each document. - `category`: Category of the document — `3gpp-standard` or `etsi-standard`. - `content`: Full text of the document. Sections are separated by ` \n `. - `metadata`: JSON string with document-level metadata. Fields vary by category. ### Data Instances An example from the 3GPP subset: ``` { "id": "3gpp_24229_Rel-17_hc0", "category": "3gpp-standard", "content": "3GPP TS 24.229 Release 17 Vhc0 \n 1 Scope \n The present document specifies...", "metadata": "{\"doc_number\": \"24229\", \"series\": \"24\", \"release\": \"17\", \"version\": \"hc0\", \"filename\": \"24229-hc0.md\", \"series_dir\": \"24_series\"}" } ``` An example from the ETSI subset: ``` { "id": "etsi_28737", "category": "etsi-standard", "content": "ETSI TS 132 371 V7.3.1 (2008-07) \n Security Management concept and requirements \n 1 Scope \n The present document specifies...", "metadata": "{\"working_group\": \"TS\", \"year\": 2008, \"deliverable_type\": \"TS\", \"version\": \"7.3.1\", \"title\": \"Security Management concept and requirements\"}" } ``` ## Sample Code ```python import json from datasets import load_dataset # Full dataset ds = load_dataset("NareshModina/TeleSpec-Data") # Single category ds = load_dataset("NareshModina/TeleSpec-Data", name="etsi-standard") ds = load_dataset("NareshModina/TeleSpec-Data", name="3gpp-standard") sample = ds["train"][0] print(f"ID: {sample['id']}\nCategory: {sample['category']}\nContent: {sample['content'][:200]}") for key, value in json.loads(sample["metadata"]).items(): print(f"{key}: {value}") ``` ## Citation If you use this dataset, please cite: ```bibtex @dataset{modina2025telespecdata, author = {Naresh Modina}, title = {TeleSpec-Data: A Telecommunications Standards Dataset for Language Model Pretraining}, year = {2025}, publisher = {Hugging Face}, url = {https://huggingface.co/datasets/NareshModina/TeleSpec-Data} } ``` Please also cite the upstream sources: ```bibtex @misc{nikbakht2024tspecllm, title = {TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications}, author = {Rasoul Nikbakht and Mohamed Benzaghta and Giovanni Geraci}, year = {2024}, eprint = {2406.01768}, archivePrefix = {arXiv}, primaryClass = {cs.NI}, url = {https://arxiv.org/abs/2406.01768} } ``` ETSI document corpus sourced from [rasoul-nikbakht/NetSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/NetSpec-LLM) (CC BY-NC 4.0).license: cc-by-nc-4.0 language: - en tags: - 电信（telecom） - 大语言模型（LLM） - 欧洲电信标准协会（ETSI） - 第三代合作伙伴计划（3GPP） - 标准（standards） - 持续预训练（continual-pretraining） task_categories: - 文本生成（text-generation） size_categories: - 10M<n<100M configs: - config_name: default data_files: - split: train path: data/*/*.parquet - config_name: 3gpp-standard data_files: - split: train path: data/3gpp-standard/*.parquet - config_name: etsi-standard data_files: - split: train path: data/etsi-standard/*.parquet # TeleSpec-Data ## 数据集概览 TeleSpec-Data 是一套源自两大权威标准组织——欧洲电信标准协会（ETSI）与第三代合作伙伴计划（3GPP）的电信标准文档数据集，专为基于电信领域知识开展大语言模型（LLM）的持续预训练而设计。 本数据集包含两个类别： - **3gpp-standard**: 包含15054份第三代合作伙伴计划（3GPP）技术规范与技术报告，覆盖第8版至第19版（更新至2025年4月），数据源自 [TSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/TSpec-LLM)。 - **etsi-standard**: 包含23248份欧洲电信标准协会（ETSI）文档，覆盖15个工作组（TS、TR、EN、ES、GS、GR、EG、ETR、ETS、GTS等），时间跨度为2000年至2024年，数据从 [NetSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/NetSpec-LLM) 的PDF语料库中提取。 **总计：38302份文档。** ## 数据集结构 ### 数据字段 - `id`: 每份文档的唯一标识符。 - `category`: 文档类别，可选值为`3gpp-standard`或`etsi-standard`。 - `content`: 文档完整文本，章节之间以` `分隔。 - `metadata`: 包含文档级元数据的JSON字符串，字段内容随文档类别不同而有所差异。 ### 数据示例 以下是3GPP子集的一条示例： { "id": "3gpp_24229_Rel-17_hc0", "category": "3gpp-standard", "content": "3GPP TS 24.229 Release 17 Vhc0 1 Scope The present document specifies...", "metadata": "{"doc_number": "24229", "series": "24", "release": "17", "version": "hc0", "filename": "24229-hc0.md", "series_dir": "24_series"}" } 以下是ETSI子集的一条示例： { "id": "etsi_28737", "category": "etsi-standard", "content": "ETSI TS 132 371 V7.3.1 (2008-07) Security Management concept and requirements 1 Scope The present document specifies...", "metadata": "{"working_group": "TS", "year": 2008, "deliverable_type": "TS", "version": "7.3.1", "title": "Security Management concept and requirements"}" } ## 示例代码 python import json from datasets import load_dataset # 完整数据集 ds = load_dataset("NareshModina/TeleSpec-Data") # 单一类别数据集 ds = load_dataset("NareshModina/TeleSpec-Data", name="etsi-standard") ds = load_dataset("NareshModina/TeleSpec-Data", name="3gpp-standard") sample = ds["train"][0] print(f"ID: {sample['id']} Category: {sample['category']} Content: {sample['content'][:200]}") for key, value in json.loads(sample["metadata"]).items(): print(f"{key}: {value}") ## 引用信息 若使用本数据集，请引用以下文献： bibtex @dataset{modina2025telespecdata, author = {Naresh Modina}, title = {TeleSpec-Data: A Telecommunications Standards Dataset for Language Model Pretraining}, year = {2025}, publisher = {Hugging Face}, url = {https://huggingface.co/datasets/NareshModina/TeleSpec-Data} } 同时请引用上游数据源： bibtex @misc{nikbakht2024tspecllm, title = {TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications}, author = {Rasoul Nikbakht and Mohamed Benzaghta and Giovanni Geraci}, year = {2024}, eprint = {2406.01768}, archivePrefix = {arXiv}, primaryClass = {cs.NI}, url = {https://arxiv.org/abs/2406.01768} } ETSI文档语料库源自 [rasoul-nikbakht/NetSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/NetSpec-LLM)（CC BY-NC 4.0协议）。