Composed Encrypted Malicious Traffic Dataset for machine learning based encrypted malicious traffic analysis.

This is a traffic dataset which contains balance size of encrypted malicious and legitimate traffic for encrypted malicious traffic detection. The dataset is a secondary csv feature data which is composed of five public traffic datasets. Our dataset is composed based on three criteria: The first criterion is to combine widely considered public datasets which contain both encrypted malicious and legitimate traffic in existing works, such as the Malwares Capture Facility Project dataset and the CICIDS-2017 dataset. The second criterion is to ensure the data balance, i.e., balance of malicious and legitimate network traffic and similar size of network traffic contributed by each individual dataset. Thus, approximate proportions of malicious and legitimate traffic from each selected public dataset are extracted by using random sampling. We also ensured that there will be no traffic size from one selected public dataset that is much larger than other selected public datasets. The third criterion is that our dataset includes both conventional devices' and IoT devices' encrypted malicious and legitimate traffic, as these devices are increasingly being deployed and are working in the same environments such as offices, homes, and other smart city settings. Based on the criteria, 5 public datasets are selected. After data pre-processing, details of each selected public dataset and the final composed dataset are shown in “Dataset Statistic Analysis Document”. The document summarized the malicious and legitimate traffic size we selected from each selected public dataset, proportions of selected traffic size from each selected public dataset with respect to the total traffic size of the composed dataset (% w.r.t the composed dataset), proportions of selected encrypted traffic size from each selected public dataset (% of selected public dataset), and total traffic size of the composed dataset. From the table, we are able to observe that each public dataset equally contributes to approximately 20% of the composed dataset, except for CICDS-2012 (due to its limited number of encrypted malicious traffic). This achieves a balance across individual datasets and reduces bias towards traffic belonging to any dataset during learning. We can also observe that the size of malicious and legitimate traffic are almost the same, thus achieving class balance. The datasets now made available were prepared aiming at encrypted malicious traffic detection. Since the dataset is used for machine learning model training, a sample of train and test sets are also provided. The train and test datasets are separated based on 1:4 and stratification is applied during data split. Such datasets can be used directly for machine or deep learning model training based on selected features.

本数据集是一款面向加密恶意流量检测任务的流量数据集，其中加密恶意流量与合法加密流量的样本规模保持均衡。本数据集为二次处理得到的CSV格式特征数据，由5个公开流量数据集整合而成。本数据集的构建遵循三项准则：第一项准则为整合现有研究中广泛使用的、同时包含加密恶意流量与合法加密流量的公开数据集，例如恶意软件捕获设施项目（Malwares Capture Facility Project）数据集与CICIDS-2017数据集。第二项准则为保障数据均衡，既需实现恶意与合法网络流量的类别均衡，同时需确保各单个入选公开数据集所贡献的网络流量规模相近。为此，我们通过随机采样从每个入选公开数据集中提取近似比例的恶意与合法流量样本，并确保任意一个入选公开数据集的流量规模均不会显著高于其他入选数据集。第三项准则为覆盖传统设备与物联网（IoT）设备的加密恶意流量与合法加密流量——随着此类设备部署量持续增长，且常部署于办公室、家庭及其他智慧城市场景中，需覆盖两类设备的流量数据。基于上述三项准则，我们最终选取了5个公开数据集。经数据预处理后，各入选公开数据集与最终整合数据集的详细信息详见《数据集统计分析文档》。该文档汇总了从各入选公开数据集中提取的恶意与合法流量规模、各入选公开数据集所选流量规模占最终整合数据集总流量的比例（相对于整合数据集的百分比，即"% w.r.t the composed dataset"）、各入选公开数据集所选加密流量规模占该数据集自身的比例（"% of selected public dataset"），以及最终整合数据集的总流量规模。从该文档的表格中可以看到，除CICDS-2012数据集（因其加密恶意流量样本量有限）外，每个入选公开数据集对整合数据集的贡献比例均约为20%。此举实现了各数据集间的均衡，降低了模型训练过程中对特定数据集流量的偏向性。同时可观察到，恶意流量与合法流量的规模几乎一致，实现了类别均衡。当前公开的本数据集专为加密恶意流量检测任务构建。鉴于本数据集用于机器学习模型训练，我们还提供了训练集与测试集的样本。训练集与测试集按照1:4的比例划分，且划分过程中采用了分层抽样策略。本数据集可直接基于所选特征，用于机器学习或深度学习模型的训练。

搜集汇总

背景与挑战

背景概述

该数据集是一个平衡的加密恶意和合法流量数据集，由五个公开数据集组合而成，确保了数据平衡和减少偏差。数据集包含113个特征，适用于机器学习模型训练，并提供了训练和测试集的划分。