An Open-Source Framework and Dataset for Multi-Layer Monitoring and Predictive Autoscaling of 6G Video Streaming

The "An Open-Source Framework and Dataset for Multi-Layer Monitoring and Predictive Autoscaling of 6G Video Streaming" dataset was created by the University of West Attica, by collecting data from a multiple MediaMTX-based video streaming scenarios for the purposes of Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation program SAFE-6G under Grant Agreement No 101139031. To ensure high observability, operational resilience, and data-driven optimization, our system integrates a robust monitoring and containerized infrastructure using Prometheus an open-source time-series database, Thanos extending Prometheus by enabling scalable, long-term storage and high availability across clusters and Kepler attributing power consumption to individual processes and Kubernetes pods, enabling the correlation of detailed energy usage data with resource utilization metrics gathered by Prometheus. In this dataset, we collected 17 metrics from 2 different memory allocation setups (100MiB, 200MiB) and 3 different scenarios (1,2 and 3 streamers) with and increasing number of streamers. The metrics range from streaming session telemetry, energy consumption and container-level resource usage. The MediaMTX core of the experiment was populated by three distinct instances (Google Colab, VM, and a local machine) with an increasing number of concurrent sessions every 30 seconds with an offset of 10 seconds to each other up to a maximum of 30. Each scenario lasted approximately 16 minutes and every metric entry was collected every second. These metrics effectively capture factors closely associated with system failure events that are critical in analyzing the scenarios. Category Metrics (Units) Definition Energy Consumption container_core_joules_total (Joules) container_dram_joules_total (Joules) container_joules_total (Joules) These metrics represent power usage at CPU and DRAM levels. Memory Metrics container_memory_cache (Bytes) container_memory_failures_total (Counter container_memory_usage_bytes (Bytes) container_memory_working_set_bytes (Bytes) container_memory_usage_bytes_ratio (Ratio) container_memory_working_set_bytes_ratio (Ratio) These provide insight into real-time memory usage, cache behavior and proportional usage within resource limits. Storage Metric container_fs_reads_bytes_total (Bytes) Tracks read operations from the file system. Streaming Metrics rtsp_sessions (Counter) rtsp_sessions_total (Counter) rtsp_sessions_bytes_sent (Bytes) colab_failed_streams_timestamps (Timestamp) vm_failed_streams_timestamps (Timestamp) local_failed_streams_timestamps (Timestamp) These metrics reflect active and cumulative session counts, as well as data throughput via the RTSP protocol. Operational Status pod_container_status_running (Boolean) This monitors the container uptime status. By identifing the moments of critical failures events where e.g. rtsp_sessions_s0, container_memory_usage_bytes_ratio_s0 (where _s0 a prefix that indicates the streamer that the metric was collected from) values decrease by a significant amount and when the number of failed streams (colab_failed_streams_timestamps, vm_failed_streams_timestamps, local_ failed_streams_timestamps) increases. We labeled the dataset by referencing these moments for each streamer (e.g. label_s0), where "1" a critical failure and "0" normal conditions. This labeled dataset can be used to train ML methods to predict the a critical system failure that can lead to e.g. loss of connections beforehand.