Emotional Genes: A Bio-Inspired Multimodal Emotion Recognition Framework via DNA Sequence Encoding and DNA-BERT

Multimodal emotion recognition (MER) infers human affect from signals such as EEG, facial expressions, speech, and text, yet remains challenged by temporal asynchrony, semantic inconsistency, and distributional heterogeneity. We propose SW-DNABERT, a framework that formulates MER as symbolic sequence modeling. It decomposes each modality into multi-scale atoms, encodes them into DNA-like sequences and employs entropy-regularized optimal transport to align sequences according to their cross-modal handshake strength, which reflects the coupling intensity between emotional patterns across modalities. The gene handshake strength is a quantitative metric describing the degree of coupling between two emotional gene units. It captures the extent to which different modalities exhibit joint activation, pattern consistency, and mutual reinforcement under the same emotional state. Inspired by biological base-pair coupling in DNA, the metric evaluates whether, for instance, an EEG pattern and a facial-expression pattern are “handshaking,” and whether the handshake is strong enough to signify a coherent emotional signal. A higher handshake strength indicates greater cross-modal consistency, synchrony, and reliability among emotional gene units, whereas a lower strength suggests cross-modal disagreement or noise. In the proposed model, the handshake strength is intrinsically linked to the entropy-regularized Optimal Transport formulation. Related Datasets The proposed implementation is evaluated on the following public benchmark datasets (all datasets are used in their heatmap-based feature representations in this study): CMU-MOSEI MELD IEMOCAP MER rPPG–EEG File Description CSV files: Contain numerical datasets composed of multi-scale extracted features. diagram/: Heatmaps generated from multimodal feature representations. DNAbert_for_classification_transition/: Source code implementing the classifier. encoder_5.zip contains multiple DNA encoding schemes designed in our study. Authorship Note Guoming Chen and Qinru Guo contributed equally to this work and are co–first authors.