SIMBED - Offline Real-World Wireless Networking Experimentation using ns-3

R&D in wireless networking typically depends on experimentation to make realistic evaluations, since simulation is inherently a simplification of the real-world. However, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility. Real wireless experiments are hardly repeatable. Given the same input they can produce very different output results, since wireless communications are influenced by external random phenomena such as noise, interference, and multipath. Real experiments are also difficult to reproduce: either the original community testbed is unavailable – offline or running other experiments – or the custom testbed used is inaccessible. Fed4FIRE+ wireless testbeds such as w-iLab.t and NITOS, although deployed in controlled environments, do not fully address the problem. The CONCRETE tool used in such testbeds assures the repeatability and reproducibility of experiments, but ignores executions whose results are also representative of the system operation and often reveal unpredicted behaviour that must be understood. What if we could make any wireless experiment repeatable and reproducible under the same exact conditions? What if we could share the same Fed4FIRE+ testbed execution conditions among an "infinite" number of users? What if we could run wireless experiments faster than in real time? INESC TEC has been developing the Offline Experimentation (OE) approach that combines the best of simulation and experimentation to achieve the above-mentioned goals. By relying on Network Simulator 3 (ns-3) and its good simulation capabilities from the MAC to the application layer, we have been exploring how ns-3 can be used to replicate real-world wireless experiments using real traces containing 1) position of nodes and 2) the quality of each radio link. The SIMBED project aimed at running a set of wireless experiments on top of the controlled environments of w-ilab.t and NITOS Fed4FIRE+ testbeds to further validate the OE approach. For that purpose, we configured different fixed and mobile experimental scenarios, representative of Wi-Fi range of operation, and measured the attained network performance using metrics such as throughput and Round-Trip Time (RTT). Then, we repeated each experiment using, both, Pure Simulation (PS) and OE approaches based on ns-3, also measuring the network performance for the same set of executions of experiments for all the different scenarios. By comparing the performance metrics of each real experiment with its PS and OE counterparts, we were able to measure the relative error of each simulation approach relatively to the real experiments, as well as the accuracy gains introduced by the OE approach when compared to the PS traditional alternative. The main results show that it is possible to repeat and reproduce real experiments in ns-3, using the OE approach, achieving closer to real performance than using the PS approach. For all the experiments performed in SIMBED, using the OE approach resulted in an average accuracy gain of 59% when comparing to the PS approach. These results were important for validating a PhD thesis contribution related to the OE approach, as well as for producing two conference papers and one journal paper. The SIMBED results increased our confidence on the accuracy of the OE approach and are envisioned to foster the adoption of the OE approach by the networking community, in complement to the use of real experimentation. The following dataset presents the results of the SIMBED project, organized in different folders, for each subset of experiments carried on: SubExp#1: Static point-to-point Wi-Fi communications using auto-rate (Minstrel) SubExp#1.1: Using w-iLab.2 (medium to high SNR scenarios) SubExp#1.2: Using w-iLab.2 (low SNR scenarios) SubExp#1.3: Using NITOS SubExp#1.4: Using w-iLab.1 (datacenter room) SubExp#2: Static point-to-point Wi-Fi communications using fixed rate SubExp#3: Mobile point-to-point Wi-Fi communications using auto-rate (Minstrel) SubExp#4: Static multiple access Wi-Fi communications using auto-rate (Minstrel) SubExp#4.1: Using w-iLab.2 (bidirectional) (medium to high SNR scenarios) SubExp#4.2: Using w-iLab.2 (bidirectional) (low SNR scenarios) SubExp#4.3: Using NITOS (bidirectional) SubExp#4.4: Using w-iLab.1 (bidirectional) SubExp#4.5: Using NITOS (2 STAs) SubExp#4.6: Using w-iLab.2 (2 STAs) SubExpExample: contains raw experimental logs, parsed data and simulation results, to show how data extracted from the nodes is processed to be compatible with the OE approach and comparable with OE and PS simulation results. Each experiment has an individual folder, named according to the date and time of the experiment and the nodes used. Inside, there’s a folder for the parsed experimental results, which contains This folder contains the details and parsed logs of the experiment, as follows: date_time.cfg – configuration details of the experiment date_time_NodeID[1]_SenderID[2]_ReceiverID[3]_FlowType[4]_Params[5].snr – logs of the Signal/Noise ratio (1 file per node/flow) date_time_NodeID_SenderID_ReceiverID_FlowType_Params.stats – logs of the packets received (1 file per node/flow) NodeID.waypoints – coordinates of the static nodes date_time_MobileNodeID.waypoints – waypoints of the mobile nodes (when applicable) The experiment’s folder also contains a folder for the simulations output with the simulations statistics files, for the multiple simulations approaches considered, as follows: date_time_NodeID_SenderID_ReceiverID_FlowType_Params.simstats – logs of the packets received (simulation) [1] ID of the node Logging node [2] ID of the Sender node [3] ID of the Receiver node [4] Flow type: Unidirectional, Bidirectional or Unidirectional with Multiple Access [5] Configurable parameters: Sender/Receiver Transmission Power and Data Rate (when applicable)

无线网络研发通常依赖实验验证以开展贴近真实场景的性能评估，因为仿真本质上是对现实世界的简化建模。然而，实验在仿真擅长的领域存在明显局限，例如可重复性与可复现性。真实无线实验几乎无法重复：即便输入条件完全一致，也可能产出差异极大的输出结果，这是因为无线通信会受到噪声、干扰、多径效应等外部随机现象的影响。真实实验同样难以复刻：要么原始社区测试床已不可用——要么已下线，要么正在运行其他实验——要么所用的定制测试床无法访问。诸如w-iLab.t与NITOS这类Fed4FIRE+无线测试床，尽管部署在受控环境中，也未能完全解决上述问题。此类测试床中使用的CONCRETE工具可保障实验的可重复性与可复现性，但忽略了那些结果同样能体现系统运行特性、且常揭示需深入理解的非预期行为的执行场景。

倘若我们能让任意无线实验在完全一致的条件下实现可重复与可复现，又当如何？倘若我们能让“无限”数量的用户共享同一Fed4FIRE+测试床的执行条件，又当如何？倘若我们能以快于实时的速度运行无线实验，又当如何？INESC TEC团队一直致力于开发离线实验（Offline Experimentation, OE）方案，融合仿真与实验的优势以达成上述目标。依托网络模拟器3（Network Simulator 3, ns-3）及其从MAC层到应用层的优秀仿真能力，我们一直在探索如何利用ns-3，通过包含以下两项内容的真实采集轨迹数据，复现真实世界的无线实验：1）节点位置信息；2）各无线链路的通信质量。

SIMBED项目旨在基于w-iLab.t与NITOS这两个Fed4FIRE+测试床的受控环境，开展一系列无线实验，以进一步验证OE方案的有效性。为此，我们配置了多组固定与移动实验场景，覆盖Wi-Fi的典型工作范围，并通过吞吐量、往返时间（Round-Trip Time, RTT）等指标衡量所得的网络性能。随后，我们分别基于纯仿真（Pure Simulation, PS）与ns-3架构的OE方案，重复开展每组实验，并针对所有不同场景下的实验执行集合，同样测量其网络性能。

通过将每组真实实验的性能指标与其PS和OE对应结果进行对比，我们得以量化每种仿真方案相对于真实实验的相对误差，以及OE方案相较于传统PS方案所带来的精度提升。主要研究结果表明，采用OE方案、依托ns-3即可实现真实实验的可重复与可复现，其所得性能结果相较于PS方案更贴近真实场景。针对SIMBED项目开展的所有实验，OE方案相较于PS方案的平均精度提升可达59%。该研究结果不仅为一项与OE方案相关的博士论文贡献提供了验证依据，还助力产出了两篇会议论文与一篇期刊论文。SIMBED的实验结果增强了我们对OE方案精度的信心，且有望推动网络社区采纳OE方案，作为真实实验的补充手段。

本数据集收录了SIMBED项目的研究成果，按实验子集划分为不同文件夹，具体如下：
SubExp#1：采用自动速率调整算法（Minstrel）的静态点对点Wi-Fi通信
SubExp#1.1：基于w-iLab.2（中高信噪比场景）
SubExp#1.2：基于w-iLab.2（低信噪比场景）
SubExp#1.3：基于NITOS
SubExp#1.4：基于w-iLab.1（数据中心机房场景）
SubExp#2：采用固定速率的静态点对点Wi-Fi通信
SubExp#3：采用自动速率调整算法（Minstrel）的移动点对点Wi-Fi通信
SubExp#4：采用自动速率调整算法（Minstrel）的静态多接入Wi-Fi通信
SubExp#4.1：基于w-iLab.2（双向传输）（中高信噪比场景）
SubExp#4.2：基于w-iLab.2（双向传输）（低信噪比场景）
SubExp#4.3：基于NITOS（双向传输）
SubExp#4.4：基于w-iLab.1（双向传输）
SubExp#4.5：基于NITOS（2个站点（Station, STA））
SubExp#4.6：基于w-iLab.2（2个站点）
SubExpExample：包含原始实验日志、解析后数据与仿真结果，用于展示如何处理从节点提取的数据，使其兼容OE方案，并可与OE及PS仿真结果进行对比。

每个实验拥有独立文件夹，以实验的日期时间与所用节点命名。文件夹内包含解析后的实验结果文件夹，其中存储了实验的详细信息与解析日志，具体如下：
date_time.cfg：实验的配置详情
date_time_NodeID[1]_SenderID[2]_ReceiverID[3]_FlowType[4]_Params[5].snr：信噪比日志（每个节点/流对应一个文件）
date_time_NodeID_SenderID_ReceiverID_FlowType_Params.stats：数据包接收日志（每个节点/流对应一个文件）
NodeID.waypoints：静态节点的坐标
date_time_MobileNodeID.waypoints：移动节点的航点（如适用）
实验文件夹内还包含一个用于存储仿真输出的文件夹，内含多种仿真方案的仿真统计文件，具体如下：
date_time_NodeID_SenderID_ReceiverID_FlowType_Params.simstats：数据包接收仿真日志
[1] 节点ID：日志记录节点
[2] 发送节点ID
[3] 接收节点ID
[4] 流类型：单向、双向或多接入单向流
[5] 可配置参数：发送/接收节点的发射功率与数据速率（如适用）