KL and CKA values for the three fusion methods.

AbstractThe objective of image fusion is to synthesize information from multiple source images into a single, high-quality composite that is information-rich, thereby enhancing both human visual interpretation and machine perception capabilities. This process also establishes a robust foundation for downstream image-related tasks. Nevertheless, current deep learning-based networks frequently neglect the distinctive features inherent in source images, presenting challenges in effectively balancing the interplay between basic and detailed features. To tackle this limitation, we introduce a progressive decomposition network that integrates Lite Transformer (LT) and ResNet architecture for infrared and visible image fusion (IVIF). Our methodology unfolds in three principal stages: Initially, a foundational convolutional neural network (CNN) is deployed to extract coarse-scale features from the source images. Subsequently, the LT is employed to bifurcate these coarse features into basic and detailed feature components. In the second phase, to augment the detail information across various inter-layer extractions, we substitute the conventional ResNet preprocessing with a combination of coarse and LT module. Cascade LT operations are implemented following the initial two ResNet blocks (ResB), enabling two-branch feature extraction from these reconfigured blocks. The final stage involves the design of specialized fusion sub-networks to process the basic and detail information blocks extracted from different layers. These processed image feature blocks are then channeled through semantic injection module (SIM) and Transformer decoders to generate the fused image. Complementing this architecture, we have developed a semantic information extraction module that aligns with the progressive inter-layer detail extraction framework. The LT module is strategically embedded within the ResNet network architecture to optimize the extraction of both basic and detailed features across diverse layers. Moreover, we introduce a novel correlation loss function that operates on the basic and detail information between layers, facilitating the correlation of basic features while maintaining the independence of detail features across layers. Through comprehensive qualitative and quantitative analyses conducted on multiple infrared-visible datasets, we demonstrate the superior potential of our proposed network for advanced visual tasks. Our network exhibits remarkable performance in detail extraction, significantly outperforming existing deep learning methodologies in this domain.

摘要 图像融合的目标是将多幅源图像中的信息合成为单幅信息丰富的高质量合成图像，以此提升人类视觉解读能力与机器感知性能。该过程同时为下游图像相关任务奠定了坚实基础。然而，当前基于深度学习的网络往往忽略源图像固有的独特特征，难以有效平衡基础特征与细节特征之间的交互关系。为解决这一局限，本文提出一种融合轻量Transformer（Lite Transformer, LT）与残差网络（ResNet）架构的渐进式分解网络，用于红外与可见光图像融合（Infrared and Visible Image Fusion, IVIF）。我们的方法主要分为三个核心阶段：首先，采用基础卷积神经网络（Convolutional Neural Network, CNN）从源图像中提取粗粒度特征；随后，借助LT将上述粗粒度特征拆解为基础特征与细节特征分量。第二阶段，为增强不同层间提取的细节信息，我们采用粗粒度模块与LT模块的组合替代传统ResNet预处理流程。在初始的两个残差块（Residual Block, ResB）之后执行级联LT操作，使重构后的残差块能够实现双分支特征提取。最后阶段，我们设计专用融合子网络以处理从不同层级提取的基础特征块与细节特征块，将处理后的图像特征块通过语义注入模块（Semantic Injection Module, SIM）与Transformer解码器进行处理，最终生成融合图像。为配合该架构，我们还开发了与渐进式层间细节提取框架相适配的语义信息提取模块。LT模块被战略性地嵌入ResNet网络架构中，以优化不同层级的基础特征与细节特征提取效果。此外，我们提出一种全新的关联损失函数，该函数作用于层间的基础信息与细节信息，在促进基础特征跨层关联的同时，保证各层细节特征的独立性。通过在多组红外-可见光数据集上开展全面的定性与定量分析，我们验证了所提网络在高级视觉任务中的优异性能潜力。该网络在细节提取方面表现出色，在该领域显著优于现有深度学习方法。