A Multimodal Dataset for Mixed Emotion Recognition

ABSTRACT: Mixed emotions have attracted an increasing interests recently, but existing datasets rarely focus on mixed emotion recognition from multimodal signals. Therefore, it is necessary to establish a dataset to support mixed emotion recognition research, and in this work, we present such a multi-modal dataset with four kind of signals recorded during watching mixed and non-mixed stimuli videos. To ensure the effect of emotion induction, we first implemented a rule-based video filtering step that filters stimuli videos based on emotion intensity. Then we conducted experiments on 35 subjects using the selected video clips. Four-kind of signals, including EEG, GSR, PPG and frontal face videos are recorded while the subjects watching 32 video clips from 4 blocks, and for each trial, we also recorded three self reports (i.e., PANAS, valence-arousal-dominance, amusement-disgust). The multimodal signal data and self-assessment data of 28 subjects together constitute the dataset. Technical validation for emotion induction and mixed emotion classification from physiological signals and face videos are also presented.

摘要：近年来，混合情绪已逐渐成为研究热点，但现有数据集极少聚焦于多模态信号下的混合情绪识别任务。因此，构建支撑混合情绪识别研究的专用数据集具有重要现实意义，本研究即公开了这样一个多模态数据集，该数据集采集了受试者观看混合情绪与非混合情绪诱发视频时的四类信号。为保障情绪诱发效果，研究团队首先实施了基于规则的视频筛选流程，依据情绪强度对候选诱发视频进行筛选。随后，研究团队使用筛选出的32个视频片段（分为4个实验区块）对35名受试者开展了实验。受试者在观看视频的过程中，同步采集了四类信号：脑电图（EEG）、皮肤电反应（GSR）、光电容积描记信号（PPG）以及正面面部视频；此外，每一轮实验还记录了三项自评数据，分别为积极与消极情感量表（PANAS）、效价-唤醒度-支配度评分以及愉悦-厌恶自评数据。最终，28名受试者的多模态信号数据与自评数据共同构成了本数据集。本研究同时提供了基于生理信号与面部视频的情绪诱发效果验证及混合情绪分类技术验证方案。