Passive Operating System Fingerprinting Revisited - Network Flows Dataset

For the evaluation of OS fingerprinting methods, we need a dataset with the following requirements: First, the dataset needs to be big enough to capture the variability of the data. In this case, we need many connections from different operating systems. Second, the dataset needs to be annotated, which means that the corresponding operating system needs to be known for each network connection captured in the dataset. Therefore, we cannot just capture any network traffic for our dataset; we need to be able to determine the OS reliably. To overcome these issues, we have decided to create the dataset from the traffic of several web servers at our university. This allows us to address the first issue by collecting traces from thousands of devices ranging from user computers and mobile phones to web crawlers and other servers. The ground truth values are obtained from the HTTP User-Agent, which resolves the second of the presented issues. Even though most traffic is encrypted, the User-Agent can be recovered from the web server logs that record every connection’s details. By correlating the IP address and timestamp of each log record to the captured traffic, we can add the ground truth to the dataset. For this dataset, we have selected a cluster of five web servers that host 475 unique university domains for public websites. The monitoring point recording the traffic was placed at the backbone network connecting the university to the Internet. The dataset used in this paper was collected from approximately 8 hours of university web traffic throughout a single workday. The logs were collected from Microsoft IIS web servers and converted from W3C extended logging format to JSON. The logs are referred to as <em>web logs </em>and are used to annotate the records generated from packet capture obtained by using a network probe tapped into the link to the Internet. The entire dataset creation process consists of seven steps: The packet capture was processed by the Flowmon flow exporter (https://www.flowmon.com) to obtain primary flow data containing information from TLS and HTTP protocols. Additional statistical features were extracted using GoFlows flow exporter (https://github.com/CN-TU/go-flows). The primary flows were filtered to remove incomplete records and network scans. The flows from both exporters were merged together into records containing fields from both sources. <em>Web logs </em>were filtered to cover the same time frame as the flow records. <em>Web logs</em> were paired with the flow records based on shared properties (IP address, port, time). The last step was to convert the User-Agent values into the operating system using a Python version of the open-source tool <em>ua-parser</em> (https://github.com/ua-parser/uap-python). We replaced the unstructured User-Agent string in the records with the resulting OS. The collected and enriched flows contain 111 data fields that can be used as features for OS fingerprinting or any other data analyses. The fields grouped by their area are listed below: basic flow properties - flow_ID;start;end;L3 PROTO;L4 PROTO;BYTES A;PACKETS A;SRC IP;DST IP;TCP flags A;SRC port;DST port;packetTotalCountforward;packetTotalCountbackward;flowDirection;flowEndReason; IP parameters - IP ToS;maximumTTLforward;maximumTTLbackward;IPv4DontFragmentforward;IPv4DontFragmentbackward; TCP parameters - TCP SYN Size;TCP Win Size;TCP SYN TTL;tcpTimestampFirstPacketbackward;tcpOptionWindowScaleforward;tcpOptionWindowScalebackward;tcpOptionSelectiveAckPermittedforward;tcpOptionSelectiveAckPermittedbackward;tcpOptionMaximumSegmentSizeforward;tcpOptionMaximumSegmentSizebackward;tcpOptionNoOperationforward;tcpOptionNoOperationbackward;synAckFlag;tcpTimestampFirstPacketforward; HTTP - HTTP Request Host;URL; User-agent - UA OS family;UA OS major;UA OS minor;UA OS patch;UA OS patch minor; TLS - TLS_CONTENT_TYPE;TLS_HANDSHAKE_TYPE;TLS_SETUP_TIME;TLS_SERVER_VERSION;TLS_SERVER_RANDOM;TLS_SERVER_SESSION_ID;TLS_CIPHER_SUITE;TLS_ALPN;TLS_SNI;TLS_SNI_LENGTH;TLS_CLIENT_VERSION;TLS_CIPHER_SUITES;TLS_CLIENT_RANDOM;TLS_CLIENT_SESSION_ID;TLS_EXTENSION_TYPES;TLS_EXTENSION_LENGTHS;TLS_ELLIPTIC_CURVES;TLS_EC_POINT_FORMATS;TLS_CLIENT_KEY_LENGTH;TLS_ISSUER_CN;TLS_SUBJECT_CN;TLS_SUBJECT_ON;TLS_VALIDITY_NOT_BEFORE;TLS_VALIDITY_NOT_AFTER;TLS_SIGNATURE_ALG;TLS_PUBLIC_KEY_ALG;TLS_PUBLIC_KEY_LENGTH;TLS_JA3_FINGERPRINT; Packet timings - NPM_CLIENT_NETWORK_TIME;NPM_SERVER_NETWORK_TIME;NPM_SERVER_RESPONSE_TIME;NPM_ROUND_TRIP_TIME;NPM_RESPONSE_TIMEOUTS_A;NPM_RESPONSE_TIMEOUTS_B;NPM_TCP_RETRANSMISSION_A;NPM_TCP_RETRANSMISSION_B;NPM_TCP_OUT_OF_ORDER_A;NPM_TCP_OUT_OF_ORDER_B;NPM_JITTER_DEV_A;NPM_JITTER_AVG_A;NPM_JITTER_MIN_A;NPM_JITTER_MAX_A;NPM_DELAY_DEV_A;NPM_DELAY_AVG_A;NPM_DELAY_MIN_A;NPM_DELAY_MAX_A;NPM_DELAY_HISTOGRAM_1_A;NPM_DELAY_HISTOGRAM_2_A;NPM_DELAY_HISTOGRAM_3_A;NPM_DELAY_HISTOGRAM_4_A;NPM_DELAY_HISTOGRAM_5_A;NPM_DELAY_HISTOGRAM_6_A;NPM_DELAY_HISTOGRAM_7_A;NPM_JITTER_DEV_B;NPM_JITTER_AVG_B;NPM_JITTER_MIN_B;NPM_JITTER_MAX_B;NPM_DELAY_DEV_B;NPM_DELAY_AVG_B;NPM_DELAY_MIN_B;NPM_DELAY_MAX_B;NPM_DELAY_HISTOGRAM_1_B;NPM_DELAY_HISTOGRAM_2_B;NPM_DELAY_HISTOGRAM_3_B;NPM_DELAY_HISTOGRAM_4_B;NPM_DELAY_HISTOGRAM_5_B;NPM_DELAY_HISTOGRAM_6_B;NPM_DELAY_HISTOGRAM_7_B; ICMP - ICMP TYPE; The details of OS distribution grouped by the OS family are summarized in the table below. The <em>Other </em>OS family contains records generated by web crawling bots that do not include OS information in the User-Agent. OS Family Number of flows Other 42474 Windows 40349 Android 10290 iOS 8840 Mac OS X 5324 Linux 1589 Ubuntu 653 Fedora 88 Chrome OS 53 Symbian OS 1 Slackware 1 Linux Mint 1为了评估操作系统指纹识别（OS fingerprinting）方法，我们需要满足以下要求的数据集：其一，数据集需具备足够规模以覆盖数据的分布差异，具体而言，需要包含来自不同操作系统的大量网络连接；其二，数据集需带有标注，即数据集中每条捕获的网络连接都需明确其对应的操作系统。因此，我们无法直接采集任意网络流量作为数据集，必须能够可靠地确定每条流量对应的操作系统。为解决上述问题，我们决定从本校多台Web服务器的流量中构建该数据集。通过采集数千台设备的流量——涵盖个人计算机、移动设备、网络爬虫及其他服务器——我们可以满足第一个要求。真实标签（ground truth）通过HTTP User-Agent字段获取，以此解决第二个问题。尽管大部分流量已加密，但我们可从记录每条连接详情的Web服务器日志中恢复User-Agent字段。通过将每条日志记录的IP地址与时间戳与捕获的流量进行关联，我们便能为数据集添加真实标签。本次研究选用了承载475个唯一高校公开网站域名的5台Web服务器集群。流量监控节点部署于高校连接互联网的骨干网络链路处。本文使用的数据集采集自单个工作日约8小时的高校Web流量。日志源自Microsoft IIS Web服务器，并从W3C扩展日志格式转换为JSON格式。这些日志被称为<em>Web日志</em>，用于标注通过接入互联网链路的网络探针捕获的数据包记录。整个数据集构建流程包含七个步骤：1. 通过Flowmon流量导出器（https://www.flowmon.com）处理数据包捕获结果，获取包含传输层安全协议（TLS）与超文本传输协议（HTTP）信息的基础流量数据。2. 使用GoFlows流量导出器（https://github.com/CN-TU/go-flows）提取额外统计特征。3. 过滤基础流量，移除不完整记录与网络扫描流量。4. 将两个导出器生成的流量合并为包含两类数据源字段的记录。5. 过滤Web日志，使其时间范围与流量记录保持一致。6. 根据共享属性（IP地址、端口、时间）将Web日志与流量记录进行匹配配对。7. 最后一步，通过开源工具<em>ua-parser</em>（https://github.com/ua-parser/uap-python）的Python版本将User-Agent值转换为对应的操作系统，将记录中非结构化的User-Agent字符串替换为识别得到的操作系统。采集并丰富后的流量数据共包含111个可作为操作系统指纹识别或其他数据分析特征的字段。按领域分组的字段如下：- 基础流量属性：flow_ID、start、end、L3 PROTO、L4 PROTO、BYTES A、PACKETS A、SRC IP、DST IP、TCP flags A、SRC port、DST port、packetTotalCountforward、packetTotalCountbackward、flowDirection、flowEndReason- IP参数：IP ToS、maximumTTLforward、maximumTTLbackward、IPv4DontFragmentforward、IPv4DontFragmentbackward- TCP参数：TCP SYN Size、TCP Win Size、TCP SYN TTL、tcpTimestampFirstPacketbackward、tcpOptionWindowScaleforward、tcpOptionWindowScalebackward、tcpOptionSelectiveAckPermittedforward、tcpOptionSelectiveAckPermittedbackward、tcpOptionMaximumSegmentSizeforward、tcpOptionMaximumSegmentSizebackward、tcpOptionNoOperationforward、tcpOptionNoOperationbackward、synAckFlag、tcpTimestampFirstPacketforward- HTTP：HTTP Request Host、URL- 用户代理（User-agent）：UA OS family、UA OS major、UA OS minor、UA OS patch、UA OS patch minor- TLS：TLS_CONTENT_TYPE、TLS_HANDSHAKE_TYPE、TLS_SETUP_TIME、TLS_SERVER_VERSION、TLS_SERVER_RANDOM、TLS_SERVER_SESSION_ID、TLS_CIPHER_SUITE、TLS_ALPN、TLS_SNI、TLS_SNI_LENGTH、TLS_CLIENT_VERSION、TLS_CIPHER_SUITES、TLS_CLIENT_RANDOM、TLS_CLIENT_SESSION_ID、TLS_EXTENSION_TYPES、TLS_EXTENSION_LENGTHS、TLS_ELLIPTIC_CURVES、TLS_EC_POINT_FORMATS、TLS_CLIENT_KEY_LENGTH、TLS_ISSUER_CN、TLS_SUBJECT_CN、TLS_SUBJECT_ON、TLS_VALIDITY_NOT_BEFORE、TLS_VALIDITY_NOT_AFTER、TLS_SIGNATURE_ALG、TLS_PUBLIC_KEY_ALG、TLS_PUBLIC_KEY_LENGTH、TLS_JA3_FINGERPRINT- 数据包时序：NPM_CLIENT_NETWORK_TIME、NPM_SERVER_NETWORK_TIME、NPM_SERVER_RESPONSE_TIME、NPM_ROUND_TRIP_TIME、NPM_RESPONSE_TIMEOUTS_A、NPM_RESPONSE_TIMEOUTS_B、NPM_TCP_RETRANSMISSION_A、NPM_TCP_RETRANSMISSION_B、NPM_TCP_OUT_OF_ORDER_A、NPM_TCP_OUT_OF_ORDER_B、NPM_JITTER_DEV_A、NPM_JITTER_AVG_A、NPM_JITTER_MIN_A、NPM_JITTER_MAX_A、NPM_DELAY_DEV_A、NPM_DELAY_AVG_A、NPM_DELAY_MIN_A、NPM_DELAY_MAX_A、NPM_DELAY_HISTOGRAM_1_A、NPM_DELAY_HISTOGRAM_2_A、NPM_DELAY_HISTOGRAM_3_A、NPM_DELAY_HISTOGRAM_4_A、NPM_DELAY_HISTOGRAM_5_A、NPM_DELAY_HISTOGRAM_6_A、NPM_DELAY_HISTOGRAM_7_A、NPM_JITTER_DEV_B、NPM_JITTER_AVG_B、NPM_JITTER_MIN_B、NPM_JITTER_MAX_B、NPM_DELAY_DEV_B、NPM_DELAY_AVG_B、NPM_DELAY_MIN_B、NPM_DELAY_MAX_B、NPM_DELAY_HISTOGRAM_1_B、NPM_DELAY_HISTOGRAM_2_B、NPM_DELAY_HISTOGRAM_3_B、NPM_DELAY_HISTOGRAM_4_B、NPM_DELAY_HISTOGRAM_5_B、NPM_DELAY_HISTOGRAM_6_B、NPM_DELAY_HISTOGRAM_7_B- ICMP：ICMP TYPE按操作系统家族分组的操作系统分布详情汇总如下。<em>Other</em>操作系统家族包含由网络爬虫生成的记录，这类爬虫的User-Agent中未包含操作系统信息。| 操作系统家族 | 流量数量 || --- | --- || Other | 42474 || Windows | 40349 || Android | 10290 || iOS | 8840 || Mac OS X | 5324 || Linux | 1589 || Ubuntu | 653 || Fedora | 88 || Chrome OS | 53 || Symbian OS | 1 || Slackware | 1 || Linux Mint | 1