Measuring Network Latency from a Wireless ISP: Variations Within and Across Subnets - Dataset

Overview

This archive contains the data from a network latency measurement campaign conducted inside the wireless ISP JackRabbit Wireless between 2023-10-30 and 2023-12-01. The measurement setup and an analysis of the data is provided in the paper "Measuring Network Latency from a Wireless ISP: Variations Within and Across Subnets", accepted for the ACM Internet Measurement Conference 2024. 

Code for parsing the data and reproducing the plots can be found at https://github.com/simosund/Measuring-Network-Latency-from-a-Wireless-ISP.git

Manifest

The raw subfolder: contains the original data captured by epping from the measurement campaign. However, note that we have merged all subnet entries from the internal network into a single entry at each timepoint to avoid revealing details about the subnet structure on the internal network. Some internal subnet entries that only carry control traffic have also been removed. It is still possible to recreate all the results from the paper with this data.

The processed subfolder: This contains processed versions of the data that have been merged in a few different ways, allowing the results from the paper to be recreated quickly without having to parse all of the original data.

subnets_stats_merged.json: The data from the entire measurement period have been merged into a single entry per subnet

{lan,wan}_per10s_stats_merged.json: All the internal or external data merged into a single entry per timepoint (10s).

top_100asn_stats_10min_json: The external data merged per ASN (for the 100 ASNs with the highest amount of downlink traffic during the measurement period) at a 10 minute resolution.

protocol_counters.hdf5: Counters for number of packets of each IP-protocol. The most common protocols, such as TCP, UDP and ICMP are named, where less common ones are just numbered.

ecn_counters.hdf5: Counters of the four ECN-marks (Not-ECT, ECT(1), ECT(0) and CE).

The overhead_measurements subfolder: CPU and network load measurements from a later period used to estimate the overhead of the epping tool. In the covered period, epping was running between the 19th of April and the 3rd of May.

InfluxDB_CPU_20240413-20240513.csv: CSV with hourly CPU utilization per core on the DuT.

jackrabbit-shaper-per\_day-20240421-20240513.csv: CSV with network load metrics (throughput in bytes and packets) every 48 minutes.

proglistoutput_after_enabling_bpf_stats_24hr.txt: Output from bpftool prog list after enabling sysctl -w kernel.bpf_stats_enabled=1, containing run counts and cumulative runtime for all eBPF programs on the system.

rrc14_20231115_ipasn.dat: Data from RIPE´s route collector RRC14 in the middle of the measurement campaign (2023-11-15), used to map subnets into ASNs.

asn.txt: RIPE's list of AS names (from https://ftp.ripe.net/ripe/asnames/asn.txt), used to translate ASNs into AS names.

Data format for raw data

The raw data is organized into one subfolder per day, with each subfolder containing one compressed json-file per minute. All files have been included in a tar-archive.

The JSON format in each file consists of a single list of entries. There are three types of entries.

Configuration entry

Each file starts with one configuration entry, containing the following fields:

 timestamp: Unix timestamp in nanoseconds when entry was created

bins: The maximum number of bins in the histogram of a subnet entry. Always 250 in this dataset. Note that histograms may contain less bins as unused bins in the upper range are truncated.

bin_width_ns: The width of the bins in nanoseconds. Always 4000000 (4ms) in this dataset.

aggregation_interval_ns: How often the aggregated stats are reported in nanoseconds. Always 10000000000 (10s) in this dataset.

timeout_interval_ns: How long epping will keep a subnet entry around since it last saw traffic for it. Always 30000000000 (30s) in this dataset. Note that a subnet entry may only be deleted after its stats are reported, so no stats will be lost regardless of value.

ipv4_prefix_len: Subnet size to aggregate IPv4 traffic. Always 24 (/24) in this dataset.

ipv6_prefix_len: Subnet size to aggregate IPv6 traffic. Always /48 in this dataset.

Example:

{"timestamp": 1698661062033443066,
"bins": 250,
"bin_width_ns": 4000000,
"aggregation_interval_ns": 10000000000,
"timeout_interval_ns": 30000000000,
"ipv4_prefix_len": 24,
"ipv6_prefix_len": 48}

Subnet stats entry

The most common entry will be the subnet stats entry, which contains the aggregated stats for one /24 subnet (or /48 if IPv6) over a 10 second period. Each entry always contains the following fields:

timestamp: A unix timestamp in nanoseconds of when the entry were reported. The entry will thus contain the stats for the period [timestamp - 10s, timestamp].

ip_prefix: The subnet in standard IPv4 or IPv6 CIDR notation.

rx_stats and tx_stats: Packet and byte counts for packets received from/transmitted to the subnet. Note that rx and tx is from the perspective of the epping node, not the perspective of the subnet, so rx is packets epping has seen sent from the subnet, and tx is packets epping has seen sent towards the subnet.

TCP_TS, TCP_noTS and other: Entries for TCP traffic that has TCP timestamps enabled, TCP traffic with TCP timestamps disabled, and non-TCP traffic, respectively. The entry will be omitted if there is no traffic of the relevant type.

packets: Number of packets of the specific type

bytes: Sum of bytes in the observed packets, including headers (up to and including the 14-byte Ethernet header).

Additionally, the following fields will be present if any RTT has been observed for the subnet. Note that the subnet will include RTTs for traffic going towards the subnet, i.e. SRC -> subnet -> SRC.

min_rtt: The minimum observed RTT in nanoseconds.

max_rtt: The maximum observed RTT in nanoseconds.

histogram:: A histogram with all observed RTTs. The bin width and maximum number of bins is specified by the configuration entry (always 4ms wide bins and max 250 bins in this dataset). Note that the histogram will truncate 0-bins after the last non-zero bin (e.g. if the maximum RTT is 7 ms, the histogram will only contain 2 bins).

count_rtt: The total number of RTTs observed (sum of histogram bin counts)

mean_rtt: The approximate mean RTT in nanoseconds.

median_rtt: The approximate median RTT in nanoseconds.

p95_rtt: The approximate 95th percentile of the RTTs in nanoseconds

Note that mean_rtt, median_rtt and p95_rtt are all calculated from the histogram, assuming all RTTs are in the middle of the histogram bins, and will thus be +/- 2ms of the real value in this dataset.

Example:

{"timestamp": 1698661072800129847,
"ip_prefix": "100.64.0.0/24",
"rx_stats": {"TCP_TS": {"packets": 64816, "bytes": 23063341},
"other": {"packets": 39783, "bytes": 16074723},
"TCP_noTS": {"packets": 8505, "bytes": 7403649}},
"tx_stats": {"TCP_TS": {"packets": 134424, "bytes": 175370347},
"other": {"packets": 76267, "bytes": 86104300},
"TCP_noTS": {"packets": 7946, "bytes": 5174210}},
"min_rtt": 50766,
"max_rtt": 440315940,
"histogram": [622,
596,
835,
1106,
932,
410,
228,
139,
108,
68,
78,
79,
73,
63,
49,
35,
22,
33,
31,
24,
21,
18,
23,
15,
20,
20,
37,
35,
96,
113,
95,
80,
36,
22,
24,
27,
17,
31,
9,
9,
14,
18,
10,
21,
22,
21,
22,
25,
8,
9,
16,
8,
12,
9,
7,
5,
6,
6,
3,
4,
4,
2,
2,
1,
1,
3,
3,
1,
0,
3,
2,
0,
1,
1,
1,
1,
2,
2,
1,
2,
1,
0,
0,
0,
1,
2,
0,
1,
1,
0,
0,
1,
0,
1,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
1],
"count_rtt": 6568,
"mean_rtt": 37529232.64311815,
"median_rtt": 18000000.0,
"p95_rtt": 150000000.0}

Global counters entry

Each report interval will also contain one entry with global counters, with the following fields:

timestamp: Unix timestamp in nanoseconds of when the entry was reported. The entry will thus contain stats for the period [timestamp - 10s, timestamp].

protocol_counters: A set of fields with packet and byte counters for the protocols indicated in the IP-header. The most common protocols will be named, specifically TCP, UDP, ICMP, ICMPv6 and the special non-IP for non-IP packets. All other protocols will simply use the protocol number. Protocols for which no traffic has been observed are omitted.

packets: The number of packets of the specific type

bytes: Sum of bytes in the observed packets, including headers (up to and including the 14-byte Ethernet header). Only the list of named protocols above will include this byte counter.

ecn_counters: Counters for the four distinct ECN code points in the IP header. May contain the fields no_ECT (00), ECT1 (01), ECT0 (10) and CE (11).

Example:

{"timestamp": 1698661072848798575,
"protocol_counters": {"TCP": {"packets": 249032, "bytes": 242282230},
"UDP": {"packets": 119778, "bytes": 108251262},
"ICMP": {"packets": 1523, "bytes": 445347},
"ICMPv6": {"packets": 87, "bytes": 25031},
"41": {"packets": 1},
"47": {"packets": 251},
"89": {"packets": 22}},
"ecn_counters": {"no_ECT": 347980, "ECT1": 2, "ECT0": 22709, "CE": 3},
"errors": {}}