Light Snowfall (DENSE)

多模态传感器流（例如摄像头、激光雷达和雷达测量）的融合在自动驾驶汽车的目标检测中发挥着关键作用，自动驾驶汽车的决策基于这些输入。虽然现有方法在良好的环境条件下利用冗余信息，但在感觉流可能不对称扭曲的恶劣天气中却失败了。这些罕见的“边缘情况”场景没有在可用的数据集中表示，现有的融合架构也不是为了处理它们而设计的。为了应对这一挑战，我们提出了一个在北欧超过 10,000 公里的驾驶中获得的新型多模式数据集。尽管该数据集是恶劣天气下的第一个大型多模态数据集，有 10 万个用于激光雷达、摄像头、雷达和门控 NIR 传感器的标签，但由于极端天气很少见，因此它不利于训练。为此，我们提出了一个用于鲁棒融合的深度融合网络，而无需覆盖所有不对称失真的大量标记训练数据。从提案级融合出发，我们提出了一个单次模型，该模型在测量熵的驱动下自适应地融合特征。
我们引入了一个对象检测数据集，用于在雾室受控天气条件下覆盖现实世界驾驶场景中的恶劣天气条件。该数据集涵盖了雾、雪和雨等多种天气条件，是在北欧超过 10,000 公里的驾驶中获得的。捕获路线和传感器设置如上所示。总共有 100k 个对象用准确的 2D 和 3D 边界框标记。以下是恶劣天气下的示例视频。

Fusion of multimodal sensor streams (e.g., camera, LiDAR, and radar measurements) plays a critical role in object detection for autonomous vehicles, whose decision-making relies on these inputs. While existing methods leverage redundant information under favorable environmental conditions, they fail in harsh weather scenarios where sensor streams may be asymmetrically distorted. These rare "edge case" scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge, we present a novel multimodal dataset collected from over 10,000 kilometers of driving in Northern Europe. As the first large-scale multimodal dataset under harsh weather conditions, it contains 100,000 labels for LiDAR, camera, radar, and gated NIR sensors. However, it is not conducive to training due to the rarity of extreme weather. To this end, we propose a deep fusion network for robust fusion without requiring large amounts of labeled training data covering all asymmetric distortions. Starting from proposal-level fusion, we introduce a one-shot model that adaptively fuses features driven by measurement entropy. We additionally present an object detection dataset that covers harsh weather conditions in real-world driving scenarios under controlled fog chamber conditions. This dataset includes various weather conditions such as fog, snow, and rain, and was collected from over 10,000 kilometers of driving in Northern Europe. The capture routes and sensor setup are shown above. In total, 100,000 objects are annotated with accurate 2D and 3D bounding boxes. Example videos under harsh weather conditions are provided below.

搜集汇总

背景与挑战

背景概述

该数据集是一个专注于自动驾驶目标检测的多模态数据集，收集自北欧超过10,000公里的驾驶场景，覆盖雾、雪和雨等多种恶劣天气条件，包含10万个带2D和3D边界框标签的对象。它是首个大型恶劣天气多模态数据集，旨在解决传感器融合在不对称失真天气中的失效问题，用于训练鲁棒的深度融合网络。