Network traffic datasets with novel extended IP flow called NetTiSA flow

Network traffic datasets with novel extended IP flow called NetTiSA flow

Datasets were created for the paper: NetTiSA: Extended IP Flow with Time-series Features for Universal Bandwidth-constrained High-speed Network Traffic Classification -- Josef Koumar, Karel Hynek, Jaroslav Pešek, Tomáš Čejka -- which is published in The International Journal of Computer and Telecommunications Networking https://doi.org/10.1016/j.comnet.2023.110147Please cite the usage of our datasets as:

Josef Koumar, Karel Hynek, Jaroslav Pešek, Tomáš Čejka, "NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification", Computer Networks, Volume 240, 2024, 110147, ISSN 1389-1286

@article{KOUMAR2024110147,
title = {NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification},
journal = {Computer Networks},
volume = {240},
pages = {110147},
year = {2024},
issn = {1389-1286},
doi = {https://doi.org/10.1016/j.comnet.2023.110147},
url = {https://www.sciencedirect.com/science/article/pii/S1389128623005923},
author = {Josef Koumar and Karel Hynek and Jaroslav Pešek and Tomáš Čejka}
}

This Zenodo repository contains 23 datasets created from 15 well-known published datasets, which are cited in the table below. Each dataset contains the NetTiSA flow feature vector. 

NetTiSA flow feature vector

The novel extended IP flow called NetTiSA (Network Time Series Analysed) flow contains a universal bandwidth-constrained feature vector consisting of 20 features. We divide the NetTiSA flow classification features into three groups by computation. The first group of features is based on classical bidirectional flow information---a number of transferred bytes, and packets.  The second group contains statistical and time-based features calculated using the time-series analysis of the packet sequences. The third type of features can be computed from the previous groups (i.e., on the flow collector) and improve the classification performance without any impact on the telemetry bandwidth.

Flow features

The flow features are:

Packets is the number of packets in the direction from the source to the destination IP address.

Packets in reverse order is the number of packets in the direction from the destination to the source IP address.

Bytes is the size of the payload in bytes transferred in the direction from the source to the destination IP address.

Bytes in reverse order is the size of the payload in bytes transferred in the direction from the destination to the source IP address.

Statistical and Time-based features

The features that are exported in the extended part of the flow. All of them can be computed (exactly or in approximative) by stream-wise computation, which is necessary for keeping memory requirements low. The second type of feature set contains the following features:

Mean represents mean of the payload lengths of packets

Min is the minimal value from payload lengths of all packets in a flow

Max is the maximum value from payload lengths of all packets in a flow

Standard deviation is a measure of the variation of payload lengths from the mean payload length

Root mean square is the measure of the magnitude of payload lengths of packets

Average dispersion is the average absolute difference between each payload length of the packet and the mean value

Kurtosis is the measure describing the extent to which the tails of a distribution differ from the tails of a normal distribution

Mean of relative times is the mean of the relative times which is a sequence defined as \(st = \{t_1 - t_1, t_2 - t_1, ..., t_n - t_1\} \)

Mean of time differences is the mean of the time differences which is a sequence defined as \(dt = \{ t_j - t_i | j = i + 1, i \in \{1, 2, \dots, n - 1\} \}.\)

Min from time differences is the minimal value from all time differences, i.e., min space between packets.

Max from time differences is the maximum value from all time differences, i.e., max space between packets.

Time distribution describes the deviation of time differences between individual packets within the time series. The feature is computed by the following equation:\(tdist = \frac{ \frac{1}{n-1} \sum_{i=1}^{n-1} \left| \mu_{\{dt_{n-1}\}} - dt_i \right| }{ \frac{1}{2} \left(max\left(\{dt_{n-1}\}\right) - min\left(\{dt_{n-1}\}\right) \right) }\)

Switching ratio represents a value change ratio (switching) between payload lengths. The switching ratio is computed by equation:\(sr = \frac{s_n}{\frac{1}{2} (n - 1)}\)

           where \(s_n\) is number of switches.

Features computed at the collectorThe third set contains features that are computed from the previous two groups prior to classification. Therefore, they do not influence the network telemetry size and their computation does not put additional load to resource-constrained flow monitoring probes. The NetTiSA flow combined with this feature set is called the Enhanced NetTiSA flow and contains the following features:

Max minus min  is the difference between minimum and maximum payload lengths

Percent deviation is the dispersion of the average absolute difference to the mean value

Variance is the spread measure of the data from its mean

Burstiness is the degree of peakedness in the central part of the distribution

Coefficient of variation is a dimensionless quantity that compares the dispersion of a time series to its mean value and is often used to compare the variability of different time series that have different units of measurement

Directions describe a percentage ratio of packet direction computed as \(\frac{d_1}{ d_1 + d_0}\), where \(d_1\) is a number of packets in a direction from source to destination IP address and \(d_0\) the opposite direction. Both  \(d_1\) and \(d_0\) are inside the classical bidirectional flow.

Duration is the duration of the flow

The NetTiSA flow is implemented into IP flow exporter ipfixprobe.

Description of dataset files

In the following table is a description of each dataset file:

File name

Detection problem

Citation of the original raw dataset

botnet_binary.csv 
Binary detection of botnet 
S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014. 

botnet_multiclass.csv 
Multi-class classification of botnet 
S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014. 

cryptomining_design.csv 
Binary detection of cryptomining; the design part 
Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022 

cryptomining_evaluation.csv 
Binary detection of cryptomining; the evaluation part 
Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022 

dns_malware.csv 
Binary detection of malware DNS 
Samaneh Mahdavifar et al. Classifying Malicious Domains using DNS Traffic Analysis. In DASC/PiCom/CBDCom/CyberSciTech 2021, pages 60–67. IEEE, 2021. 

doh_cic.csv 
Binary detection of DoH 
Mohammadreza MontazeriShatoori et al. Detection of doh tunnels using time-series classification of encrypted traffic. In DASC/PiCom/CBDCom/CyberSciTech 2020, pages 63–70. IEEE, 2020 

doh_real_world.csv 
Binary detection of DoH 
Kamil Jeřábek et al. Collection of datasets with DNS over HTTPS traffic. Data in Brief, 42:108310, 2022 

dos.csv 
Binary detection of DoS 
Nickolaos Koroniotis et al. Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: Bot-IoT dataset. Future Gener. Comput. Syst., 100:779–796, 2019. 

edge_iiot_binary.csv 
Binary detection of IoT malware 
Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022. 

edge_iiot_multiclass.csv 
Multi-class classification of IoT malware 
Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022. 

https_brute_force.csv 
Binary detection of HTTPS Brute Force 
Jan Luxemburk et al. HTTPS Brute-force dataset with extended network flows, November 2020 

ids_cic_binary.csv 
Binary detection of intrusion in IDS 
Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018. 

ids_cic_multiclass.csv 
Multi-class classification of intrusion in IDS 
Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018. 

unsw_binary.csv 
Binary detection of intrusion in IDS 
Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015. 

unsw_multiclass.csv 
Multi-class classification of intrusion in IDS 
Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015. 

iot_23.csv 
Binary detection of IoT malware 
Sebastian Garcia et al. IoT-23: A labeled dataset with malicious and benign IoT network traffic, January 2020. More details here https://www.stratosphereips.org /datasets-iot23 

ton_iot_binary.csv 
Binary detection of IoT malware 
Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021 

ton_iot_multiclass.csv 
Multi-class classification of IoT malware 
Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021 

tor_binary.csv 
Binary detection of TOR 
Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017. 

tor_multiclass.csv 
Multi-class classification of TOR 
Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017. 

vpn_iscx_binary.csv 
Binary detection of VPN 
Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016. 

vpn_iscx_multiclass.csv 
Multi-class classification of VPN 
Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016. 

vpn_vnat_binary.csv 
Binary detection of VPN 
Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022 

vpn_vnat_multiclass.csv 
Multi-class classification of VPN 
Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022

本数据集为一类包含新型扩展IP流（NetTiSA flow）的网络流量数据集。

本数据集配套论文为《NetTiSA: 面向通用带宽受限高速网络流量分类的带时序特征扩展IP流》，作者为Josef Koumar、Karel Hynek、Jaroslav Pešek、Tomáš Čejka，发表于《国际计算机与电信网络期刊》（*The International Journal of Computer and Telecommunications Networking*），DOI链接：https://doi.org/10.1016/j.comnet.2023.110147。数据集引用格式如下：

Josef Koumar, Karel Hynek, Jaroslav Pešek, Tomáš Čejka, "NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification", Computer Networks, Volume 240, 2024, 110147, ISSN 1389-1286

@article{KOUMAR2024110147,
title = {NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification},
journal = {Computer Networks},
volume = {240},
pages = {110147},
year = {2024},
issn = {1389-1286},
doi = {https://doi.org/10.1016/j.comnet.2023.110147},
url = {https://www.sciencedirect.com/science/article/pii/S1389128623005923},
author = {Josef Koumar and Karel Hynek and Jaroslav Pešek and Tomáš Čejka}
}

本Zenodo仓库包含23个由15个知名公开数据集衍生得到的数据集，原始数据集的引用信息见下表。每个数据集均包含NetTiSA流特征向量。

### NetTiSA流特征向量
新型扩展IP流NetTiSA（Network Time Series Analysed，即网络时序分析流）包含通用带宽受限特征向量，共计20项特征。我们依据计算方式将NetTiSA流分类特征划分为三类：第一类特征基于经典双向流信息，包括传输字节数与数据包数；第二类特征包含通过数据包序列时序分析计算得到的统计与时序特征；第三类特征可由前两类特征计算得到（可在流采集器侧完成），可在不影响遥测带宽的前提下提升分类性能。

### 流特征
流特征如下：
- 数据包数（Packets）：源IP地址向目的IP地址方向传输的数据包总数。
- 反向数据包数（Packets in reverse order）：目的IP地址向源IP地址方向传输的数据包总数。
- 字节数（Bytes）：源IP地址向目的IP地址方向传输的有效载荷总字节数。
- 反向字节数（Bytes in reverse order）：目的IP地址向源IP地址方向传输的有效载荷总字节数。

### 统计与时序特征
该类特征为流扩展部分所导出的特征，所有特征均可通过流逐段计算（精确或近似计算）以降低内存占用。第二类特征集包含以下特征：
- 均值（Mean）：流内所有数据包有效载荷长度的平均值。
- 最小值（Min）：流内所有数据包有效载荷长度的最小值。
- 最大值（Max）：流内所有数据包有效载荷长度的最大值。
- 标准差（Standard deviation）：数据包有效载荷长度相对于均值的离散程度。
- 均方根（Root mean square）：数据包有效载荷长度的幅值度量。
- 平均离散度（Average dispersion）：单个数据包有效载荷长度与均值的绝对差的平均值。
- 峰度（Kurtosis）：描述分布尾部与正态分布尾部差异程度的统计量。
- 相对时间均值（Mean of relative times）：相对时间序列的均值，相对时间序列定义为 \(st = \{t_1 - t_1, t_2 - t_1, ..., t_n - t_1\} \)。
- 时间差均值（Mean of time differences）：时间差序列的均值，时间差序列定义为 \(dt = \{ t_j - t_i | j = i + 1, i \in \{1, 2, \dots, n - 1\} \}.\)
- 最小时间差（Min from time differences）：所有时间差中的最小值，即数据包间的最小间隔。
- 最大时间差（Max from time differences）：所有时间差中的最大值，即数据包间的最大间隔。
- 时间分布（Time distribution）：时序内数据包间时间差的偏差程度，该特征通过以下公式计算：
\[tdist = \frac{ \frac{1}{n-1} \sum_{i=1}^{n-1} \left| \mu_{\{dt_{n-1}\}} - dt_i \right| }{ \frac{1}{2} \left(max\left(\{dt_{n-1}\}\right) - min\left(\{dt_{n-1}\}\right) \right) }\]
- 切换比率（Switching ratio）：有效载荷长度的数值变化比率，计算公式为：
\[sr = \frac{s_n}{\frac{1}{2} (n - 1)}\]
其中 \(s_n\) 为有效载荷长度发生切换的次数。

### 采集器侧计算特征
第三类特征集由前两类特征衍生得到，用于分类任务。该类特征不会影响网络遥测带宽，且不会为资源受限的流监测探针带来额外计算负载。结合该特征集的NetTiSA流称为增强型NetTiSA流，包含以下特征：
- 极差（Max minus min）：有效载荷长度最大值与最小值的差值。
- 百分偏差（Percent deviation）：平均绝对离散度相对于均值的离散程度。
- 方差（Variance）：数据相对于均值的离散程度度量。
- 突发性（Burstiness）：分布中心区域的尖峰程度。
- 变异系数（Coefficient of variation）：无量纲统计量，用于比较时序序列相对于均值的离散程度，常用于对比不同量纲时序序列的变异性。
- 方向占比（Directions）：数据包方向的百分比比率，计算公式为 \(\frac{d_1}{ d_1 + d_0}\)，其中 \(d_1\) 为源IP到目的IP方向的数据包数，\(d_0\) 为反向数据包数，二者均属于经典双向流范畴。
- 流持续时长（Duration）：流的持续时间。

NetTiSA流已在IP流导出器ipfixprobe中实现。

### 数据集文件说明
各数据集文件的说明如下表所示：

| 文件名 | 检测任务 | 原始数据集引用 |
| --- | --- | --- |
| botnet_binary.csv | 僵尸网络二元分类检测 | S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014. |
| botnet_multiclass.csv | 僵尸网络多分类检测 | S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014. |
| cryptomining_design.csv | 加密货币挖矿二元分类检测（设计集） | Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022 |
| cryptomining_evaluation.csv | 加密货币挖矿二元分类检测（评估集） | Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022 |
| dns_malware.csv | 恶意DNS二元分类检测 | Samaneh Mahdavifar et al. Classifying Malicious Domains using DNS Traffic Analysis. In DASC/PiCom/CBDCom/CyberSciTech 2021, pages 60–67. IEEE, 2021. |
| doh_cic.csv | DoH二元分类检测 | Mohammadreza MontazeriShatoori et al. Detection of doh tunnels using time-series classification of encrypted traffic. In DASC/PiCom/CBDCom/CyberSciTech 2020, pages 63–70. IEEE, 2020 |
| doh_real_world.csv | DoH二元分类检测 | Kamil Jeřábek et al. Collection of datasets with DNS over HTTPS traffic. Data in Brief, 42:108310, 2022 |
| dos.csv | DoS二元分类检测 | Nickolaos Koroniotis et al. Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: Bot-IoT dataset. Future Gener. Comput. Syst., 100:779–796, 2019. |
| edge_iiot_binary.csv | IoT恶意软件二元分类检测 | Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022. |
| edge_iiot_multiclass.csv | IoT恶意软件多分类检测 | Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022. |
| https_brute_force.csv | HTTPS暴力破解二元分类检测 | Jan Luxemburk et al. HTTPS Brute-force dataset with extended network flows, November 2020 |
| ids_cic_binary.csv | IDS入侵二元分类检测 | Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018. |
| ids_cic_multiclass.csv | IDS入侵多分类检测 | Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018. |
| unsw_binary.csv | IDS入侵二元分类检测 | Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015. |
| unsw_multiclass.csv | IDS入侵多分类检测 | Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015. |
| iot_23.csv | IoT恶意软件二元分类检测 | Sebastian Garcia et al. IoT-23: A labeled dataset with malicious and benign IoT network traffic, January 2020. More details here https://www.stratosphereips.org /datasets-iot23 |
| ton_iot_binary.csv | IoT恶意软件二元分类检测 | Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021 |
| ton_iot_multiclass.csv | IoT恶意软件多分类检测 | Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021 |
| tor_binary.csv | TOR流量二元分类检测 | Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017. |
| tor_multiclass.csv | TOR流量多分类检测 | Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017. |
| vpn_iscx_binary.csv | VPN流量二元分类检测 | Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016. |
| vpn_iscx_multiclass.csv | VPN流量多分类检测 | Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016. |
| vpn_vnat_binary.csv | VPN流量二元分类检测 | Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022 |
| vpn_vnat_multiclass.csv | VPN流量多分类检测 | Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022 |