DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset

Beam management is a challenging task for millimeter wave (mmWave) and sub-terahertz communication systems, especially in scenarios with highly-mobile users. Leveraging external sensing modalities such as vision, LiDAR, radar, position, or a combination of them, to address this beam management challenge has recently attracted increasing interest from both academia and industry. This is mainly motivated by the dependency of the beam direction decision on the user location and the geometry of the surrounding environment---information that can be acquired from the sensory data. To realize the promised beam management gains, such as the significant reduction in beam alignment overhead, in practice, however, these solutions need to account for important aspects. For example, these multi-modal sensing-aided beam selection approaches should be able to generalize their learning to unseen scenarios and should be able to operate in realistic dense deployments. In order to facilitate the generalizability study, we offer a large-scale multi-modal dataset with co-existing communication and sensing data collected across different real-world locations and different times of the day. The objective of this dataset is to enable necessary research in this direction, paving the way toward generalizable multi-modal sensing-aided beam management for real-world future communication systems.