Deep Nonparametric Bayesian Multimodal Sensor Fusion Method for Real-Time Motion and Emotion Modeling in Immersive Virtual Reali

    Immersive virtual reality (VR) is widely applied in healthcare, education and entertainment. Its key challenge is to simulate realistic environments and capture users\u2019 motion and emotional responses in real time. Existing research typically employs machine learning methods on single-modal or simple multimodal signals for emotion recognition, but single modalities are susceptible to noise and fail to reflect genuine emotions. Moreover, many works treat motion modeling and emotion recognition as separate tasks, ignoring the inherent coupling between them. This paper proposes a deep nonparametric Bayesian multimodal sensor fusion method for real-time modeling of users\u2019 motion and emotional states in immersive VR. At the sensor acquisition layer, multimodal signals (EEG, EMG, EDA, heart rate, respiration, fNIRS, etc.) are collected and a nonparametric Bayesian process adaptively learns the latent distribution of motion\u2013emotion states. At the feature fusion layer, a deep neural network is introduced to model nonlinear relationships and inference is performed using a dynamic Bayesian network. Experiments constructed immersive VR scenarios based on the DEAP dataset and compared nine baseline methods, including support vector machine, random forest, K\u2011nearest neighbors, XGBoost, CNN\u2013LSTM, residual convolutional network, BiGRU, Transformer fusion and multi\u2011head attention fusion. Results demonstrate that the proposed model significantly outperforms the comparative models in accuracy, F1 score, convergence speed and training time. The contribution of this work lies in proposing a unified nonparametric Bayesian framework for joint motion and emotion modeling, demonstrating its effectiveness in real\u2011time immersive VR environments, and providing complete algorithmic derivations and reproduction details.