Emotion Recognition for Affective human digital twin by means of virtual reality enabling technologies

We introduce a new bimodal dataset recorded during affect elicitation by means of audio-visual stimuli for human emotion recognition based on facial and corporal expressions. Our dataset was collected using three devices: an RGB camera, Kinect 1, and Kinect 2. The Kinect 1 and Kinect 2 sensors provide 121 and 1347 face key points, respectively, offering a more comprehensive analysis of facial expressions. Additionally, for the 2D RGB sequences, we utilized the feature points provided by the open-source OpenFace, which includes 2D 68 facial landmarks. From these landmarks, we selected 26 facial points that were most relevant for our emotion recognition task. To gather the data, we conducted experiments involving 17 participants. We captured both facial and skeleton keypoints, allowing for a comprehensive understanding of the participants' emotional expressions. By combining the RGB and RGB-D data from the various devices, our dataset provides a rich and diverse set of information for human emotion recognition research. This new dataset not only expands the available resources for studying human emotions but also offers a more detailed analysis with the increased number of facial keypoints provided by the Kinect sensors. Researchers can leverage this dataset to develop and evaluate more accurate and robust models for human emotion recognition, ultimately advancing our understanding of how emotions are expressed through facial and corporal cues. Please cite as: K. Amara, O. Kerdjidj and N. Ramzan, "Emotion Recognition for Affective human digital twin by means of virtual reality enabling technologies," in IEEE Access, doi: 10.1109/ACCESS.2023.3285398. Please state your name, contact details (e-mail), institution, and position, as well as the reason for requesting access to our database. For additional info contact: [e-mail] or [e-mail] [e-mail] [e-mail]本研究提出一种全新双模态数据集（bimodal dataset），该数据集通过视听刺激诱发情绪，用于基于面部与肢体表情的人类情绪识别任务。本数据集通过三类设备采集：RGB相机、Kinect 1与Kinect 2传感器。其中Kinect 1与Kinect 2可分别输出121个和1347个面部关键点，能够实现更全面的面部表情分析。此外，针对2D RGB序列，本研究采用开源工具OpenFace（OpenFace）提供的特征点，其中包含68个2D面部地标（facial landmarks）；我们从中筛选出26个与情绪识别任务相关性最高的面部关键点。 数据采集阶段共招募17名参与者开展实验，同步捕获面部与骨骼关键点，以全面解析参与者的情绪表达。通过融合多设备采集的RGB与RGB-D数据，本数据集为人类情绪识别研究提供了丰富多元的信息支撑。 该新型数据集不仅扩充了人类情绪研究的现有资源，还借助Kinect传感器提供的大量面部关键点实现了更精细化的分析。研究者可依托本数据集开发并评估更精准、鲁棒的人类情绪识别模型，最终深化对情绪通过面部与肢体线索表达机制的理解。 请按以下格式引用： K. Amara、O. Kerdjidj与N. Ramzan，《基于虚拟现实赋能技术的情感型人类数字孪生情绪识别》，发表于IEEE Access，DOI: 10.1109/ACCESS.2023.3285398。 请注明您的姓名、联系方式（电子邮箱）、所属机构与职位，以及申请使用本数据库的原因。 如需获取更多信息，请联系： [e-mail] 或 [e-mail] [e-mail] [e-mail]