MAD: A Multimodal Affective Dataset

In this study, A Multimodal Affective (MAD) containing six physiological signals and visual information was constructed to address two key issues in multimodal emotion recognition, namely, the analysis of physiological signal complementarity and the fusion of multiview visual information.The dataset was acquired from 18 subjects, including electroencephalography (EEG), electrocardiography (ECG), electro-oculography (EOG), electromyography (EMG), baroclinic shock (BCG), photoplethysmography (PPG) signals and synchronously acquired three-view RGB-D facial image data. A whole new system of labeling was constructed, containing cognitive labels (subjectively assessed by the subjects), multimedia labels (emotional labels of stimulus materials calibrated by the experimenter), and performance labels (discriminated by specific emotional performance of the subjects' faces), to provide a multidimensional basis for investigating the effects of different labels on the classification of emotions. The experimental validation showed that (1) the physiological signals were confirmed to be complementary through the feature-level fusion strategy, and the similarity of the three heart rate signals, ECG, PPG, and BCG, in emotion recognition was also verified. (2) There was a significant correlation between cognitive labels and physiological responses, which could be further considered as a subjective cognitive influence on the generalizability of the model. (3) The multiview RGB-D data fusion had an improved accuracy compared to single-view facial recognition, which verified that multiview face recognition accuracy was has been improved, which verifies the significance of the effect of multi-view emotion recognition. This study provides a novel data benchmark and fusion paradigm for multimodal emotion recognition, and the related findings are of great reference value for improving the accuracy of emotion computation in integrated multimodal wearable devices.