SMART - Self-adaptive Machine Learning Approach for Real-time Tuning of IEEE 802.11 PHY and MAC layers

Introduction Worldwide the demand for wireless access networks providing very high throughputs has been increasing exponentially, namely due to bandwidth-hungry applications such as high definition video streaming and augmented reality. In order to fulfil these requirements, the Wi-Fi standard was enriched with new amendments, such as IEEE 802.11n, IEEE 802.11ac, and recently IEEE 802.11ax (Wi-Fi 6). New parameters have been proposed for both physical (PHY) and media access control (MAC) layers, including channel bonding, short guard interval (SGI), and advanced modulation and coding schemes (MCS). However, the high variability of the signal strength in the wireless radio channel, allied to the channel asymmetry, makes the selection of optimal configurations for these parameters a challenge. Typically, these parameters are configured with a default value. For runtime optimization, some algorithms have already been proposed. Still, they were designed considering legacy IEEE 802.11 releases and static scenarios. Besides, these parameters have their trade-offs that need to be properly managed. To help dealing with this, machine learning has been recently introduced in wireless networks, providing the intelligence that networks need in order to be smart and self-adaptive. SWOP (Smart Wireless Optimization) is a cross-layer optimization approach for Wi-Fi networks extending the current Rate Adaptation (RA) approach, for instance, followed by the well-known Minstrel algorithm widely used in practice. Our approach takes advantage of Deep Reinforcement Learning (DRL) in order to learn the optimal Wi-Fi link configuration. By considering the wireless channel as the environment, the transmitter node (the agent) chooses the best link parameters (the action) in order to maximize the throughput (the reward) based on the channel metrics captured from the environment (the state). In this work we propose a simple DRL-based Wi-Fi Rate Adaptation (RA) algorithm, named Data-driven Algorithm for Rate Adaptation (DARA), which is one of the modules of Smart Wireless Optimization (SWOP) SMART aimed to run a set of wireless experiments on top of w-iLab.t testbeds provided by the Fed4FIRE+ project to directly validate our DRL model and learn a policy from the wireless experiments executed in a controlled environment. However, after facing difficulties with the scenarios we could achieve on the real testbed, we decided to train and test DARA using a trace-based simulation approach. In simulation, we could train our model in scenarios that are more complex and diverse whilst easy to configure, when compared to real testbeds. The w-iLab.t testbeds were still used to capture data traces (e.g. Signal-to-Noise Ratio, position of nodes, transmission power and link distance) that were then injected in ns-3 for validating DARA. With this work, we concluded that DARA performance is impacted when operating in scenarios with asymmetric links, which is common in the highly dynamic and unpredictable wireless environments. Furthermore, the asymmetry offset varies between scenarios and it may also change for the same scenario, as time progresses. This randomness is not addressed when solely considering the SNR as the link metric, posing a challenge in the learning phase of DARA. Despite these limitations, the results obtained show that DARA still achieves up to 14.9% higher throughput higher than Minstrel [1] and slightly lower than Ideal [2] for most of the scenarios. The results obtained will serve as a basis to support our ongoing and future research. Folder Organization The following dataset presents the results of the SMART project, organized in different folders for each Rate Adaptation Algorithm, as well as the traces that were used to obtain such results: DARA: Results obtained using our solution (Naming Convention #1, Folder Content #1) MIN: Results obtained using Minstrel-HT (Naming Convention #1, Folder Content #2) ID: Results obtained using Ideal (Naming Convention #1, Folder Content #2) TRACES: Trace files used to obtain the results present in this dataset (Naming Convention #2, Folder Content #3) Naming Convention #1 – RAA TID TP TO: Rate Adaptation Algorithm (RAA) drl – Data Driven Algorithm for Rate Adaptation min – MinstrelHTWifiManager id – IdealWifiManager Trace ID (TID) 3 up to 8 Transport Protocol (TP) udp – User Datagram Protocol Traffic Orientation (TO) normal – A->B reversed – B->A Naming Convention #2 – TID_TXP: Trace ID (TID) 3 up to 8 Transmitting Power in dBm (TXP) 3, 5, 7, 9, 12 dBm Folder Content #1: checkpoint_ RAA TID TP TO (Folder) Policy Checkpoint with which the results were obtained Flowmonitor output for the configured scenario Column 1 – Step Counter Column 2 – Reward Value Column 3 – Observation Value Column 4 – Action Value Column 1 – Simulation Time (seconds) Column 2 – Throughput (Mbit/100ms) Folder Content #2: Flowmonitor output for the configured scenario Column 1 – Simulation Time (seconds) Column 2 – Throughput (Mbit/100ms) Folder Content #3 - Source: https://zenodo.org/record/3713271#.YjjBVDXLdhE: · date_time.cfg configuration details of the experiment · date_time_NodeID[1]_SenderID[2]_ReceiverID[3]_FlowType[4]_Params[5].snr – logs of the Signal/Noise ratio (1 file per node/flow) · date_time_NodeID_SenderID_ReceiverID_FlowType_Params.stats – logs of the packets received (1 file per node/flow) [1] ID of the node Logging node [2] ID of the Sender node [3] ID of the Receiver node [4] Flow type: Unidirectional, Bidirectional or Unidirectional with Multiple Access [5] Configurable parameters: Sender/Receiver Transmission Power and Data Rate (when applicable) References 1. F. FietKau, “Minstrel_HT: New rate control module for 802.11n [LWN.net]”. Mrt-2010. 2. “ns-3: ns3::IdealWifiManager Class Reference,” Jan 2021, [Online; accessed 23. Jun. 2021]. Available: https://www.nsnam.org/docs/release/3.33/doxygen/classns3_1_1_ideal_wifi manager.html

引言
当今全球范围内，支持超高吞吐量的无线接入网需求呈指数级增长，这主要源于高清视频流、增强现实等带宽密集型应用的普及。为满足此类需求，Wi-Fi标准新增了多项修订案，包括IEEE 802.11n、IEEE 802.11ac以及最新的IEEE 802.11ax（Wi-Fi 6）。研究人员已为物理层（PHY，Physical Layer）与媒体访问控制层（MAC，Media Access Control）提出了多项新参数，涵盖信道绑定、短保护间隔（SGI，Short Guard Interval）以及高级调制编码方案（MCS，Advanced Modulation and Coding Schemes）。然而，无线信道中信号强度的高度可变性与信道非对称性，使得为上述参数选择最优配置成为一项挑战。通常情况下，这类参数会采用默认值进行配置。

针对运行时优化，已有多项算法被提出，但这些算法均基于传统IEEE 802.11版本与静态场景设计。此外，这些参数存在各自的权衡取舍，需要进行妥善管理。为应对这一问题，近年来机器学习被引入无线网络，为网络提供实现智能化与自适应所需的智能能力。

SWOP（智能无线优化，Smart Wireless Optimization）是一种面向Wi-Fi网络的跨层优化方法，它扩展了当前的速率自适应（RA，Rate Adaptation）方法——例如广泛应用于实际场景的经典Minstrel算法。本方法借助深度强化学习（DRL，Deep Reinforcement Learning）来学习最优的Wi-Fi链路配置：将无线信道视为环境，发射节点（智能体，agent）根据从环境中捕获的信道指标（状态，state），选择最优的链路参数（动作，action）以最大化吞吐量（奖励，reward）。

本研究提出了一种基于深度强化学习的简易Wi-Fi速率自适应算法，命名为数据驱动型速率自适应算法（DARA，Data-driven Algorithm for Rate Adaptation），它是智能无线优化（SWOP）SMART项目的模块之一，旨在通过Fed4FIRE+项目提供的w-iLab.t测试床开展一系列无线实验，以直接验证我们的深度强化学习模型，并从受控环境下的无线实验中学习策略。然而，在实际测试床中遇到可用场景的限制后，我们决定采用基于跟踪的仿真方法来训练与测试DARA。相较于真实测试床，仿真环境可在更易于配置的前提下，支持在更复杂、更多样的场景中训练模型。

我们仍使用w-iLab.t测试床采集数据跟踪信息（例如信噪比、节点位置、发射功率与链路距离），并将这些信息注入ns-3仿真器以验证DARA。通过本研究，我们发现DARA的性能在非对称链路场景中会受到影响——这类场景在高度动态且不可预测的无线环境中十分常见。此外，非对称偏移量在不同场景间存在差异，且在同一场景中也会随时间推移发生变化。若仅将信噪比作为链路指标，将无法应对这种随机性，这为DARA的学习阶段带来了挑战。尽管存在上述局限，实验结果表明，在大多数场景中，DARA的吞吐量仍比Minstrel[1]高出最高达14.9%，且略低于理想算法[2]。本次研究结果将作为我们当前与未来研究的基础。

文件夹组织说明
本数据集包含SMART项目的实验结果，按照不同速率自适应算法划分为多个文件夹，同时附带了用于生成上述结果的跟踪数据文件：
- DARA：使用本研究提出的解决方案得到的结果（命名约定#1，文件夹内容#1）
- MIN：使用Minstrel-HT得到的结果（命名约定#1，文件夹内容#2）
- ID：使用理想算法得到的结果（命名约定#1，文件夹内容#2）
- TRACES：用于生成本数据集结果的跟踪文件（命名约定#2，文件夹内容#3）

### 命名约定#1 – RAA TID TP TO
其中：
- RAA：速率自适应算法（Rate Adaptation Algorithm）
- drl：数据驱动型速率自适应算法
- min：MinstrelHTWifiManager
- id：IdealWifiManager
- TID：跟踪ID（Trace ID），取值范围为3至8
- TP：传输协议（Transport Protocol）
- udp：用户数据报协议（UDP，User Datagram Protocol）
- TO：流量方向（Traffic Orientation）
- normal：A→B单向流量
- reversed：B→A反向单向流量

### 命名约定#2 – TID_TXP
- TID：跟踪ID（Trace ID），取值范围为3至8
- TXP：发射功率（Transmitting Power，单位为dBm），取值为3、5、7、9、12 dBm

### 文件夹内容#1：checkpoint_RAA_TID_TP_TO（文件夹）
- 用于生成结果的模型检查点
- 对应配置场景的Flowmonitor输出：
列1：步数计数器
列2：奖励值
列3：观测值
列4：动作值
列1：仿真时间（单位：秒）
列2：吞吐量（单位：Mbit/100ms）

### 文件夹内容#2：对应配置场景的Flowmonitor输出
列1：仿真时间（单位：秒）
列2：吞吐量（单位：Mbit/100ms）

### 文件夹内容#3：数据来源：https://zenodo.org/record/3713271#.YjjBVDXLdhE
- date_time.cfg：实验配置详情
- date_time_NodeID[1]_SenderID[2]_ReceiverID[3]_FlowType[4]_Params[5].snr：信噪比日志（每个节点/流量对应一个文件）
- date_time_NodeID_SenderID_ReceiverID_FlowType_Params.stats：数据包接收日志（每个节点/流量对应一个文件）
注：
[1] 节点ID：记录日志的节点
[2] 发送节点ID
[3] 接收节点ID
[4] 流量类型：单向、双向或多址接入单向
[5] 可配置参数：发送/接收端发射功率与数据速率（如适用）

### 参考文献
1. F. FietKau，"Minstrel_HT: New rate control module for 802.11n [LWN.net]"，Mrt-2010。
2. "ns-3: ns3::IdealWifiManager Class Reference"，2021年1月，[在线；访问时间：2021年6月23日]。可获取：https://www.nsnam.org/docs/release/3.33/doxygen/classns3_1_1_ideal_wifi_manager.html