(MVRS) The Multimodal Virtual Reality Stimuli-based Emotion Recognition Dataset

<b>A</b>utomatic emotion recognition gained significant importance in the recent decade, especially with the development of artificial intelligence, which has affected our daily lives. Using personalized emotion recognition in healthcare, education, retail, and automotive has high importance these days, which requires proper data in different modalities. On the other hand, data scarcity in some emotion recognition modalities, such as body motion and physiological signals, is vivid, especially when it comes to multimodality. Furthermore, the way a participant is provoked to show emotion is crucial, as it should resemble real-life emotional expression. To do so, one of the most effective methods is to employ Virtual Reality (VR) videos and games. This paper introduces a novel Multimodal Virtual Reality Stimuli-based emotion recognition dataset, or MVRS, which could address the mentioned data scarcity issue. Our dataset contains 13 subjects or participant stimuli using VR videos for relaxation, fear, stress, sadness, and joy emotions. The dataset covers an age range of 12 to 60 in both genders. This dataset is recorded in a small lab in which all participants followed the same data collection protocols and filled out both questionnaires and consent forms. The dataset includes eye tracking, body motion, ElectroMyoGraphy (EMG), and Galvanic Skin Response (GSR) data in various formats. The eye-tracking data is recorded using a Full High-Definition (FHD) webcam placed manually into the VR Head-Mounted Display (HMD). The body motion data is recorded using Microsoft Kinect version 2. Finally, EMG and GSR data are recorded by an Arduino UNO board. All data is recorded simultaneously, with synchronized timestamps, to support clean data for multimodal processing. For each modality, related features are extracted and fused by multimodal fusion techniques (early and late stages) and evaluated using different classifiers and metrics to check the validity and separability of the data.