Multi-spectral Object Detection

多光谱图像对可以提供组合信息，使对象检测应用在开放世界中更加可靠和健壮。为了充分利用不同的模态，本文提出了一种简单而有效的跨模态特征融合方法，称为跨模态融合变压器 (CFT)。与以前基于CNNs的工作不同，我们的网络在Transformer方案的指导下，在特征提取阶段学习远程依赖关系并集成全局上下文信息。更重要的是，通过利用变压器的自我关注，网络可以自然地同时执行模态内和模态间融合，并可靠地捕获RGB和热域之间的潜在相互作用。从而大大提高了多光谱目标检测的性能。在多个数据集上进行的大量实验和消融研究表明，该方案是有效的，并获得了最先进的检测性能。我们的代码和模型可在https://github.com/DocF/多光谱-对象检测。

Multispectral image pairs can provide complementary information, making object detection applications more reliable and robust in open-world scenarios. To fully exploit distinct modalities, this paper proposes a simple yet effective cross-modal feature fusion method named Cross-modal Fusion Transformer (CFT). Unlike previous CNN-based works, our network, guided by the Transformer architecture, learns long-range dependencies and integrates global contextual information during the feature extraction phase. More importantly, by leveraging the self-attention mechanism of Transformers, the network can naturally perform intra-modal and inter-modal fusion simultaneously, and reliably capture the latent interactions between RGB and thermal domains. This significantly improves the performance of multispectral object detection. Extensive experiments and ablation studies conducted on multiple datasets demonstrate that our proposed scheme is effective and achieves state-of-the-art detection performance. Our code and models are available at https://github.com/DocF/multispectral-object-detection.

AI搜集汇总

背景与挑战

背景概述

该数据集专注于多光谱目标检测，通过结合RGB和热模态图像，提升对象检测在开放世界中的可靠性和健壮性。它采用基于Transformer的跨模态融合方法（CFT），有效集成全局上下文信息并捕获模态间相互作用，适用于自动驾驶领域。数据集由清华大学于2022年发布，包含6.9GB数据，支持GPL-3.0许可证。