Enhancing Robustness in Multimodal Emotion Recognition with State-Space Models

Abstract\u2014Deep neural networks have achieved impressive results in multimodal emotion recognition (MER), driving progress in human-computer interaction and affective computing. However, their vulnerability to adversarial perturbations remains a major obstacle to real-world deployment, as even subtle disturbances can severely degrade performance. To address this issue, we propose a robust and efffcient MER framework based on a Sinkhorn Alignment Fusion State Space Module (SAFSSM) architecture. This architecture enhances adversarial robustness and improves multimodal feature learning by capturing long-range dependencies and boosting the discriminative power of heterogeneous features. The framework incorporates heatmap-guided feature extraction and Sinkhorn normalization to softly align facial, speech, and textual features, enabling coherent and collaborative multimodal fusion. Extensive experiments show that the proposed model signiffcantly outperforms baseline methods under adversarial attacks such as FGSM, BIM, and PGD, with recognition accuracy improvements. Moreover, the proposed framework exhibits high energy efffciency and low computational overhead. We further extend multimodal emotion recognition to multi-view intelligent cockpit scenarios, demonstrating its suitability for real-time deployment in resourceconstrained environments.