A Multimodal Dataset for Mixed Emotion Recognition

ABSTRACT: Mixed emotions have attracted an increasing interests recently, but existing datasets rarely focus on mixed emotion recognition from multimodal signals. Therefore, it is necessary to establish a dataset to support mixed emotion recognition research, and in this work, we present such a multi-modal dataset with four kind of signals recorded during watching mixed and non-mixed stimuli videos. To ensure the effect of emotion induction, we first implemented a rule-based video filtering step that filters stimuli videos based on emotion intensity. Then we conducted experiments on 35 subjects using the selected video clips. Four-kind of signals, including EEG, GSR, PPG and frontal face videos are recorded while the subjects watching 32 video clips from 4 blocks, and for each trial, we also recorded three self reports (i.e., PANAS, valence-arousal-dominance, amusement-disgust). The multimodal signal data and self-assessment data of 28 subjects together constitute the dataset. Technical validation for emotion induction and mixed emotion classification from physiological signals and face videos are also presented.