Learning Modality Advantage Hierarchically for RGB-Infrared Object Detection

Background:Multi-modal object detection aims to exploit complementary cues across spectra, yet most studies focus on “how to fuse” while ignoring “what to fuse”. Empirical observations indicate that instance-level modality preference is highly heterogeneous: pedestrians exhibit stronger thermal signatures, whereas vehicles carry richer texture in the visible spectrum. Treating all features equally therefore dilutes discriminative cues and caps performance.
Methods: We propose a Hierarchical Modality-Advantage Learning (HMAL) framework that converts prior knowledge of spectrum-specific strengths into explicit learning signals. 1. Low-level: A Triple-Stream Collaborative Detection Encoder (TCDE) maintains dedicated RGB and infrared branches to preserve modality-specific details, while an auxiliary Modality-Aware Fusion (MAF) unit performs cross-spectrum alignment for detail enhancement. 2. High-level: A Modality-Advantage Guided Learning (MAGL) module first quantifies the per-category spectrum preference with a learnable advantage matrix, then injects the obtained advantage embedding into semantic features via channel-wise re-calibration. 3. Cross-level: A lightweight cross-attention block couples advantage-guided semantics with detail features, enabling complementary information to flow bidirectionally. The whole network is trained end-to-end with a composite loss that balances detection accuracy and modality-advantage consistency.
Results: Comprehensive evaluations on LLVIP, M3FD and FLIR verify that HMAL consistently surpasses existing RGB-IR detectors. The framework attains leading mAP scores while introducing negligible extra parameters and latency. Ablation studies reveal that explicit advantage modeling is the major performance driver, and visualization shows that the network automatically strengthens thermal cues for pedestrians and RGB cues for vehicles, confirming the interpretability of the proposed hierarchy. Implementation and code are available at: https://github.com/echo9958/HMAL.