(MVRS) The Multimodal Virtual Reality Stimuli-based Emotion Recognition Dataset

Automatic emotion recognition gained significant importance in the recent decade, especially with the development of artificial intelligence, which has affected our daily lives. Using personalized emotion recognition in healthcare, education, retail, and automotive has high importance these days, which requires proper data in different modalities. On the other hand, data scarcity in some emotion recognition modalities, such as body motion and physiological signals, is vivid, especially when it comes to multimodality. Furthermore, the way a participant is provoked to show emotion is crucial, as it should resemble real-life emotional expression. To do so, one of the most effective methods is to employ Virtual Reality (VR) videos and games. This paper introduces a novel Multimodal Virtual Reality Stimuli-based emotion recognition dataset, or MVRS, which could address the mentioned data scarcity issue. Our dataset contains 13 subjects or participant stimuli using VR videos for relaxation, fear, stress, sadness, and joy emotions. The dataset covers an age range of 12 to 60 in both genders. This dataset is recorded in a small lab in which all participants followed the same data collection protocols and filled out both questionnaires and consent forms. The dataset includes eye tracking, body motion, ElectroMyoGraphy (EMG), and Galvanic Skin Response (GSR) data in various formats. The eye-tracking data is recorded using a Full High-Definition (FHD) webcam placed manually into the VR Head-Mounted Display (HMD). The body motion data is recorded using Microsoft Kinect version 2. Finally, EMG and GSR data are recorded by an Arduino UNO board. All data is recorded simultaneously, with synchronized timestamps, to support clean data for multimodal processing. For each modality, related features are extracted and fused by multimodal fusion techniques (early and late stages) and evaluated using different classifiers and metrics to check the validity and separability of the data.

近十年来，自动情绪识别的重要性与日俱增，尤其是在深刻改变我们日常生活的人工智能技术快速发展的背景下。如今，在医疗健康、教育、零售与汽车等领域应用个性化情绪识别具有极高的应用价值，而这类应用亟需多模态的高质量数据支撑。另一方面，部分情绪识别模态（如人体运动与生理信号）的数据匮乏问题尤为凸显，在多模态场景下这一问题更为显著。此外，诱发受试者产生情绪的方式至关重要，该方式需贴近真实生活中的情绪表达场景。对此，采用虚拟现实（Virtual Reality, VR）视频与游戏是最为有效的手段之一。
本文提出了一款全新的基于虚拟现实诱发刺激的多模态情绪识别数据集——MVRS，可有效解决前述的数据匮乏问题。本数据集共纳入13名受试者，每位受试者均接受了针对放松、恐惧、紧张、悲伤与愉悦五种情绪的VR视频诱发刺激。受试者的年龄覆盖12至60岁区间，且涵盖男女两种性别。本数据集采集于一间小型实验室，所有受试者均遵循统一的数据采集规程，并填写了问卷与知情同意书。
本数据集包含多种格式的眼动追踪、人体运动、肌电图（ElectroMyoGraphy, EMG）以及皮肤电反应（Galvanic Skin Response, GSR）数据。其中，眼动追踪数据通过一台手动安装于虚拟现实头戴式显示器（VR Head-Mounted Display, HMD）内的全高清（Full High-Definition, FHD）网络摄像头采集；人体运动数据通过微软Kinect v2设备采集；肌电图与皮肤电反应数据则通过Arduino UNO开发板采集。所有数据均采用同步时间戳实现同步采集，可为多模态处理提供规范无噪的高质量数据。
针对每一种模态，研究人员均提取了相关特征，并通过早期融合与晚期融合等多模态融合技术进行特征融合，随后采用多种分类器与评价指标对融合结果进行评估，以验证数据集的有效性与数据可分性。