pcy12345BSU/CIC-IDS-2017

--- license: apache-2.0 task_categories: - text-classification - tabular-classification size_categories: - 100M<n<1B tags: - Network Intrusion Detection - Cybersecurity - Network Packets - CIC-IDS2017 --- We have developed a Python package as a wrapper around Hugging Face Hub and Hugging Face Datasets library to access this dataset easily. # NIDS Datasets The `nids-datasets` package provides functionality to download and utilize specially curated and extracted datasets from the original UNSW-NB15 and CIC-IDS2017 datasets. These datasets, which initially were only flow datasets, have been enhanced to include packet-level information from the raw PCAP files. The dataset contains both packet-level and flow-level data for over 230 million packets, with 179 million packets from UNSW-NB15 and 54 million packets from CIC-IDS2017. ## Installation Install the `nids-datasets` package using pip: ```shell pip install nids-datasets ``` Import the package in your Python script: ```python from nids_datasets import Dataset, DatasetInfo ``` ## Dataset Information The `nids-datasets` package currently supports two datasets: [UNSW-NB15](https://research.unsw.edu.au/projects/unsw-nb15-dataset) and [CIC-IDS2017](https://www.unb.ca/cic/datasets/ids-2017.html). Each of these datasets contains a mix of normal traffic and different types of attack traffic, which are identified by their respective labels. The UNSW-NB15 dataset has 10 unique class labels, and the CIC-IDS2017 dataset has 24 unique class labels. - UNSW-NB15 Labels: 'normal', 'exploits', 'dos', 'fuzzers', 'generic', 'reconnaissance', 'worms', 'shellcode', 'backdoor', 'analysis' - CIC-IDS2017 Labels: 'BENIGN', 'FTP-Patator', 'SSH-Patator', 'DoS slowloris', 'DoS Slowhttptest', 'DoS Hulk', 'Heartbleed', 'Web Attack – Brute Force', 'Web Attack – XSS', 'Web Attack – SQL Injection', 'Infiltration', 'Bot', 'PortScan', 'DDoS', 'normal', 'exploits', 'dos', 'fuzzers', 'generic', 'reconnaissance', 'worms', 'shellcode', 'backdoor', 'analysis', 'DoS GoldenEye' ## Subsets of the Dataset Each dataset consists of four subsets: 1. Network-Flows - Contains flow-level data. 2. Packet-Fields - Contains packet header information. 3. Packet-Bytes - Contains packet byte information in the range (0-255). 4. Payload-Bytes - Contains payload byte information in the range (0-255). Each subset contains 18 files (except Network-Flows, which has one file), where the data is stored in parquet format. In total, this package provides access to 110 files. You can choose to download all subsets or select specific subsets or specific files depending on your analysis requirements. ## Getting Information on the Datasets The `DatasetInfo` function provides a summary of the dataset in a pandas dataframe format. It displays the number of packets for each class label across all 18 files in the dataset. This overview can guide you in selecting specific files for download and analysis. ```python df = DatasetInfo(dataset='UNSW-NB15') # or dataset='CIC-IDS2017' df ``` ## Downloading the Datasets The `Dataset` class allows you to specify the dataset, subset, and files that you are interested in. The specified data will then be downloaded. ```python dataset = 'UNSW-NB15' # or 'CIC-IDS2017' subset = ['Network-Flows', 'Packet-Fields', 'Payload-Bytes'] # or 'all' for all subsets files = [3, 5, 10] # or 'all' for all files data = Dataset(dataset=dataset, subset=subset, files=files) data.download() ``` The directory structure after downloading files: ``` UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ └───Payload-Bytes ├───Payload_Bytes_File_3.parquet ├───Payload_Bytes_File_5.parquet └───Payload_Bytes_File_10.parquet ``` You can then load the parquet files using pandas: ```python import pandas as pd df = pd.read_parquet('UNSW-NB15/Packet-Fields/Packet_Fields_File_10.parquet') ``` ## Merging Subsets The `merge()` method allows you to merge all data of each packet across all subsets, providing both flow-level and packet-level information in a single file. ```python data.merge() ``` The merge method, by default, uses the details specified while instantiating the `Dataset` class. You can also pass subset=list of subsets and files=list of files you want to merge. The directory structure after merging files: ``` UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ ├───Payload-Bytes │ ├───Payload_Bytes_File_3.parquet │ ├───Payload_Bytes_File_5.parquet │ └───Payload_Bytes_File_10.parquet │ └───Network-Flows+Packet-Fields+Payload-Bytes ├───Network_Flows+Packet_Fields+Payload_Bytes_File_3.parquet ├───Network_Flows+Packet_Fields+Payload_Bytes_File_5.parquet └───Network_Flows+Packet_Fields+Payload_Bytes_File_10.parquet ``` ## Extracting Bytes Packet-Bytes and Payload-Bytes subset contains the first 1500-1600 bytes. To retrieve all bytes (up to 65535 bytes) from the Packet-Bytes and Payload-Bytes subsets, use the `Bytes()` method. This function requires files in the Packet-Fields subset to operate. You can specify how many bytes you want to extract by passing the max_bytes parameter. ```python data.bytes(payload=True, max_bytes=2500) ``` Use packet=True to extract packet bytes. You can also pass files=list of files to retrieve bytes. The directory structure after extracting bytes: ``` UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ ├───Payload-Bytes │ ├───Payload_Bytes_File_3.parquet │ ├───Payload_Bytes_File_5.parquet │ └───Payload_Bytes_File_10.parquet │ ├───Network-Flows+Packet-Fields+Payload-Bytes │ ├───Network_Flows+Packet_Fields+Payload_Bytes_File_3.parquet │ ├───Network_Flows+Packet_Fields+Payload_Bytes_File_5.parquet │ └───Network_Flows+Packet_Fields+Payload_Bytes_File_10.parquet │ └───Payload-Bytes-2500 ├───Payload_Bytes_File_3.parquet ├───Payload_Bytes_File_5.parquet └───Payload_Bytes_File_10.parquet ``` ## Reading the Datasets The `read()` method allows you to read files using Hugging Face's `load_dataset` method, one subset at a time. The dataset and files parameters are optional if the same details are used to instantiate the `Dataset` class. ```python dataset = data.read(dataset='UNSW-NB15', subset='Packet-Fields', files=[1,2]) ``` The `read()` method returns a dataset that you can convert to a pandas dataframe or save to a CSV, parquet, or any other desired file format: ```python df = dataset.to_pandas() dataset.to_csv('file_path_to_save.csv') dataset.to_parquet('file_path_to_save.parquet') ``` For scenarios where you want to process one packet at a time, you can use the `stream=True` parameter: ```python dataset = data.read(dataset='UNSW-NB15', subset='Packet-Fields', files=[1,2], stream=True) print(next(iter(dataset))) ``` ## Notes The size of these datasets is large, and depending on the subset(s) selected and the number of bytes extracted, the operations can be resource-intensive. Therefore, it's recommended to ensure you have sufficient disk space and RAM when using this package.--- 许可证：Apache-2.0 任务类别： - 文本分类 - 表格分类 数据规模： - 100M < n < 1B 标签： - 网络入侵检测 - 网络安全 - 网络数据包 - CIC-IDS2017 --- 我们开发了一款封装了Hugging Face 枢纽 (Hugging Face Hub) 与Hugging Face 数据集库 (Hugging Face Datasets library) 的Python软件包，以实现该数据集的便捷访问。 # 网络入侵检测数据集（NIDS Datasets） `nids-datasets` 软件包支持下载并使用从原始UNSW-NB15与CIC-IDS2017数据集精心整理提取得到的子数据集。上述两类数据集最初仅包含流级数据，现已通过原始PCAP文件 (Packet Capture file) 补充了数据包级信息。本数据集总计涵盖超过2.3亿条数据包的数据包级与流级数据，其中UNSW-NB15数据集包含1.79亿条数据包，CIC-IDS2017数据集包含5400万条数据包。 ## 安装方式 使用pip安装`nids-datasets`软件包： shell pip install nids-datasets 在Python脚本中导入该软件包： python from nids_datasets import Dataset, DatasetInfo ## 数据集详情 `nids-datasets` 软件包目前支持两类数据集：[UNSW-NB15](https://research.unsw.edu.au/projects/unsw-nb15-dataset) 与 [CIC-IDS2017](https://www.unb.ca/cic/datasets/ids-2017.html)。每类数据集均包含正常流量与多种攻击流量，通过对应标签进行区分。UNSW-NB15数据集包含10种唯一类别标签，CIC-IDS2017数据集包含24种唯一类别标签。 - UNSW-NB15 类别标签：'normal'（正常流量）、'exploits'（漏洞利用攻击）、'dos'（拒绝服务攻击）、'fuzzers'（模糊测试攻击）、'generic'（通用攻击）、'reconnaissance'（侦察攻击）、'worms'（蠕虫攻击）、'shellcode'（Shell代码攻击）、'backdoor'（后门攻击）、'analysis'（分析探测攻击） - CIC-IDS2017 类别标签：'BENIGN'（正常流量）、'FTP-Patator'（FTP暴力破解）、'SSH-Patator'（SSH暴力破解）、'DoS slowloris'（Slowloris拒绝服务）、'DoS Slowhttptest'（Slowhttptest拒绝服务）、'DoS Hulk'（Hulk拒绝服务）、'Heartbleed'（心脏滴血漏洞）、'Web Attack – Brute Force'（Web暴力破解攻击）、'Web Attack – XSS'（跨站脚本攻击（XSS））、'Web Attack – SQL Injection'（SQL注入攻击）、'Infiltration'（渗透入侵）、'Bot'（僵尸网络）、'PortScan'（端口扫描）、'DDoS'（分布式拒绝服务攻击）、'normal'（正常流量）、'exploits'（漏洞利用攻击）、'dos'（拒绝服务攻击）、'fuzzers'（模糊测试攻击）、'generic'（通用攻击）、'reconnaissance'（侦察攻击）、'worms'（蠕虫攻击）、'shellcode'（Shell代码攻击）、'backdoor'（后门攻击）、'analysis'（分析探测攻击）、'DoS GoldenEye'（GoldenEye拒绝服务攻击） ## 数据集子集 每类数据集均包含4个子集： 1. 网络流（Network-Flows）：包含流级数据。 2. 数据包字段（Packet-Fields）：包含数据包头部信息。 3. 数据包字节（Packet-Bytes）：包含取值范围为0~255的数据包字节信息。 4. 负载字节（Payload-Bytes）：包含取值范围为0~255的负载字节信息。 除网络流子集仅包含1个文件外，其余每个子集均包含18个文件，所有数据均以Parquet格式存储。本软件包总计提供110个文件的访问权限，用户可根据分析需求选择下载全部子集、指定子集或指定文件。 ## 获取数据集信息 `DatasetInfo` 函数可返回以Pandas库 (Pandas) 数据框格式呈现的数据集概览，展示数据集中全部18个文件内各分类标签对应的数据包数量，该概览可辅助用户选择需下载与分析的特定文件。 python df = DatasetInfo(dataset='UNSW-NB15') # 或 dataset='CIC-IDS2017' df ## 下载数据集 `Dataset` 类支持用户指定所需的数据集、子集与文件，随后将自动下载对应数据。 python dataset = 'UNSW-NB15' # 或 'CIC-IDS2017' subset = ['Network-Flows', 'Packet-Fields', 'Payload-Bytes'] # 或 'all' 以下载全部子集 files = [3, 5, 10] # 或 'all' 以下载全部文件 data = Dataset(dataset=dataset, subset=subset, files=files) data.download() 下载文件后的目录结构： UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ └───Payload-Bytes ├───Payload_Bytes_File_3.parquet ├───Payload_Bytes_File_5.parquet └───Payload_Bytes_File_10.parquet 随后可通过Pandas库加载Parquet文件： python import pandas as pd df = pd.read_parquet('UNSW-NB15/Packet-Fields/Packet_Fields_File_10.parquet') ## 合并子集 `merge()` 方法可将所有子集内的单数据包数据进行合并，最终生成同时包含流级与数据包级信息的单一文件。 python data.merge() 该方法默认使用实例化`Dataset`类时指定的参数，用户也可手动传入`subset`（子集列表）与`files`（文件列表）参数以指定需合并的内容。 合并文件后的目录结构： UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ ├───Payload-Bytes │ ├───Payload_Bytes_File_3.parquet │ ├───Payload_Bytes_File_5.parquet │ └───Payload_Bytes_File_10.parquet │ └───Network-Flows+Packet-Fields+Payload-Bytes ├───Network_Flows+Packet_Fields+Payload_Bytes_File_3.parquet ├───Network_Flows+Packet_Fields+Payload_Bytes_File_5.parquet └───Network_Flows+Packet_Fields+Payload_Bytes_File_10.parquet ## 提取字节数据 数据包字节与负载字节子集默认仅包含前1500~1600字节的数据。若需从上述两个子集提取全部字节（最多可达65535字节），可使用`Bytes()`方法。该方法需依赖数据包字段子集的文件方可运行，用户可通过`max_bytes`参数指定需提取的字节数。 python data.bytes(payload=True, max_bytes=2500) 可通过传入`packet=True`参数提取数据包字节，也可指定`files`参数以选择需提取字节的文件。 提取字节数据后的目录结构： UNSW-NB15 │ ├───Network-Flows │ └───UNSW_Flow.parquet │ ├───Packet-Fields │ ├───Packet_Fields_File_3.parquet │ ├───Packet_Fields_File_5.parquet │ └───Packet_Fields_File_10.parquet │ ├───Payload-Bytes │ ├───Payload_Bytes_File_3.parquet │ ├───Payload_Bytes_File_5.parquet │ └───Payload_Bytes_File_10.parquet │ ├───Network-Flows+Packet-Fields+Payload-Bytes │ ├───Network_Flows+Packet_Fields+Payload_Bytes_File_3.parquet │ ├───Network_Flows+Packet_Fields+Payload_Bytes_File_5.parquet │ └───Network_Flows+Packet_Fields+Payload_Bytes_File_10.parquet │ └───Payload-Bytes-2500 ├───Payload_Bytes_File_3.parquet ├───Payload_Bytes_File_5.parquet └───Payload_Bytes_File_10.parquet ## 读取数据集 `read()` 方法可通过Hugging Face的`load_dataset`方法读取文件，每次读取一个子集。若实例化`Dataset`类时已指定相关参数，则`dataset`与`files`参数为可选参数。 python dataset = data.read(dataset='UNSW-NB15', subset='Packet-Fields', files=[1,2]) `read()` 方法返回的数据集可转换为Pandas数据框，或保存为CSV、Parquet等任意目标格式： python df = dataset.to_pandas() dataset.to_csv('file_path_to_save.csv') dataset.to_parquet('file_path_to_save.parquet') 若需逐数据包处理数据，可使用`stream=True`参数： python dataset = data.read(dataset='UNSW-NB15', subset='Packet-Fields', files=[1,2], stream=True) print(next(iter(dataset))) ## 注意事项 本数据集体量较大，根据所选子集与提取的字节数不同，相关操作可能占用较多系统资源。因此，使用本软件包时，请确保具备足够的磁盘空间与运行内存。