CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360 VR Videos

CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360° Videos ## General Information We develop the CEAP-360VR dataset to address the lack of continuously annotated behavioral and physiological datasets for 360 video VR affective computing. Accordingly, this dataset contains a) questionnaires (SSQ, IPQ, NASA-TLX); b) continuous valence-arousal annotations; c) head and eye movements as well as left and right eye pupil diameters while watching videos; d) peripheral physiological responses (ACC, EDA, SKT, BVP, HR, IBI). Our dataset also concludes the data pre-processing, data validating scripts, along with dataset description and key steps in the stage of data acquisition and pre-processing. ## Dataset Structure The CEAP-360VR folder contains the following six subfolders 1_Stimuli 2_QuestionnaireData 3_AnnotationData 4_BehaviorData 5_PhysioData 6_Scripts The following is a detailed description of each sub-file: 1_Stimuli - VideoThumbNails contains the eight thumbNails for each video (.jpg) - VideoInfo.json contains the detailed information for eight videos 2_QuestionnaireData - PXX_Questionnaire_Data.json (X = 1, 2, ..., 32) contains questionnaire data for each participant 3_AnnotationData - Raw contains the raw annotation data captured from the Joy-Con joystick for each participant - Transformed contains the transformed valence-arousal data generated from the raw data for each participant - Frame contains the re-sampled annotation data from the transformed data for each participant 4_BehaviorData - Raw contains the raw behavior data captured from the HTC VIVE Pro Eye Tobii Device for each participant - Transformed contains the transformed heam/eye movement data (pitch/yaw) generated from the raw data, as well as pupil diameter data for each participant - Frame contains the re-sampled behavior data generated from the transformed data for each participant - HM_ScanPath contains the head scanpath data generated from the transformed data for each participant - EM_Fixation contains the eye gaze fixation data generated from the transformed data for each participant 5_PhysioData - Raw contains the raw physiological data captured from the Empatica E4 wristband for each participant - Transformed contains the transformed physiological data generated from the raw data for each participant - Frame contains the re-sampled physiological data from the transformed data for each participant 6_Scripts - Unity Project contains the complete project of our user-controlled experiment (Unity 2018.4.1f1, HTC VIVE Pro Eye HMD) - Data Processed contains scripts that undertake the pre-processing steps for converting the raw data to the transformed/frame data in the transformed and frame folders. conatins scripts for continuous annotation, behavior and physiological data analysis and visualization. - CEAP-360VR_Baseline contains scripts to generate processed behavioral and physiological data with V-A labels for deep learning experiments and features for machine learning experiments. contains scripts to run ML and DL experiments under both subject-dependent and subject-independent model. ## Dataset Description The CEAP-360VR Dataset [Description.pdf](https://github.com/cwi-dis/CEAP-360VR-Dataset/blob/master/CEAP-Dataset%20Description.pdf) introduces the dataset description and key steps in the stage of data acquisition and pre-processing. ## Dataset License CEAP-360VR dataset is licensed under a [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license]. ## Citation Please cite our paper in any published work that uses this dataset as follows: - Plain Text T. Xue, A. El Ali, T. Zhang, G. Ding, and P. Cesar, "CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360° Videos," in IEEE Transactions on Multimedia, doi: 10.1109/TMM.2021.3124080. - BibTex @ARTICLE{Xue2021CEAP-360VR, author={Xue, Tong and Ali, Abdallah El and Zhang, Tianyi and Ding, Gangyi and Cesar, Pablo}, journal={IEEE Transactions on Multimedia}, title={CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360° Videos}, year={2021}, volume={}, number={}, pages={1-1}, doi={https://doi.org/10.1109/TMM.2021.3124080}} ## Usage 1. We have performed the time alignment of different types of data and videos for each participant, as well as the proceesing scripts that can be used to generate both the transformed and frame data. Researchers can run their analysis methods on them. 2. For researchers who want to try other data processing methods, you can directly use the raw data. ## About The CEAP-360VR Dataset is maintained by Key Laboratory of Digital Performance and Simulation Technology at Beijing Institute of Technology and the Distributed & Interactive Systems (DIS) research group at Centrum Wiskunde & Informatica . Contact the authors - Tong Xue: [e-mail], [e-mail] - Abdallah El Ali: [e-mail]

CEAP-360VR：面向360°视频的连续生理与行为情感标注数据集

## 通用信息
为解决360°视频虚拟现实（VR, Virtual Reality）情感计算领域缺乏连续标注的行为与生理数据集的问题，我们构建了CEAP-360VR数据集。本数据集包含以下四类内容：a）问卷调查量表（SSQ、IPQ、NASA-TLX）；b）连续的效价-唤醒（valence-arousal）标注；c）观看视频过程中的头部与眼球运动数据，以及左右眼瞳孔直径数据；d）外周生理响应数据（ACC、EDA、SKT、BVP、HR、IBI）。此外，本数据集还包含数据预处理、数据验证脚本，以及数据集说明文档与数据采集、预处理阶段的关键步骤说明。

## 数据集结构
CEAP-360VR文件夹包含以下6个子文件夹：
1. 1_Stimuli（刺激素材文件夹）
2. 2_QuestionnaireData（问卷调查数据文件夹）
3. 3_AnnotationData（标注数据文件夹）
4. 4_BehaviorData（行为数据文件夹）
5. 5_PhysioData（生理数据文件夹）
6. 6_Scripts（脚本文件夹）

以下为各子文件夹的详细说明：
### 1_Stimuli
- VideoThumbNails：包含每个视频的8张缩略图（格式为JPEG(.jpg)）
- VideoInfo.json：包含8个视频的详细信息

### 2_QuestionnaireData
- PXX_Questionnaire_Data.json（X=1,2,…,32）：包含每位被试的问卷调查数据

### 3_AnnotationData
- Raw：包含每位被试通过Joy-Con摇杆采集的原始标注数据
- Transformed：包含从原始数据转换得到的、每位被试的效价-唤醒转换数据
- Frame：包含从转换后数据重采样得到的、每位被试的帧级标注数据

### 4_BehaviorData
- Raw：包含每位被试通过HTC VIVE Pro Eye眼动仪采集的原始行为数据
- Transformed：包含从原始数据转换得到的、每位被试的头部/眼球运动数据（俯仰角/偏航角），以及瞳孔直径数据
- Frame：包含从转换后数据重采样得到的、每位被试的帧级行为数据
- HM_ScanPath：包含从转换后数据生成的、每位被试的头部扫描路径数据
- EM_Fixation：包含从转换后数据生成的、每位被试的眼动注视点数据

### 5_PhysioData
- Raw：包含每位被试通过Empatica E4腕带采集的原始生理数据
- Transformed：包含从原始数据转换得到的、每位被试的转换后生理数据
- Frame：包含从转换后数据重采样得到的、每位被试的帧级生理数据

### 6_Scripts
- Unity Project：包含我们的用户可控实验的完整项目（基于Unity 2018.4.1f1、HTC VIVE Pro Eye头戴显示器（HMD, Head-Mounted Display））
- Data Processed：包含用于将原始数据转换为转换文件夹与帧文件夹中的转换后数据、帧数据的预处理脚本；同时包含用于连续标注、行为与生理数据分析及可视化的脚本
- CEAP-360VR_Baseline：包含用于生成带效价-唤醒标签的处理后行为与生理数据的脚本（用于深度学习实验），以及用于机器学习（ML, Machine Learning）实验的特征生成脚本；同时包含用于在被试内、被试间两种模型下运行机器学习与深度学习（DL, Deep Learning）实验的脚本

## 数据集说明
CEAP-360VR数据集的说明文档（[Description.pdf](https://github.com/cwi-dis/CEAP-360VR-Dataset/blob/master/CEAP-Dataset%20Description.pdf)）介绍了数据集详情与数据采集、预处理阶段的关键步骤。

## 数据集许可证
CEAP-360VR数据集采用[知识共享署名-非商业性使用4.0国际许可协议（Creative Commons Attribution-NonCommercial 4.0 International, CC BY-NC 4.0）]进行授权。

## 引用
若您的研究工作使用了本数据集，请按以下格式引用本论文：
- 纯文本格式
T. Xue, A. El Ali, T. Zhang, G. Ding, and P. Cesar, "CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360° Videos," in IEEE Transactions on Multimedia, doi: 10.1109/TMM.2021.3124080.
- BibTex格式
@ARTICLE{Xue2021CEAP-360VR,
author={Xue, Tong and Ali, Abdallah El and Zhang, Tianyi and Ding, Gangyi and Cesar, Pablo},
journal={IEEE Transactions on Multimedia},
title={"CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360° Videos"},
year={2021},
volume={},
number={},
pages={1-1},
doi={https://doi.org/10.1109/TMM.2021.3124080}}

## 使用方法
1. 我们已为每位被试完成了不同类型数据与视频的时间对齐工作，并提供了可用于生成转换后数据与帧级数据的处理脚本，研究人员可基于此运行自定义分析方法。
2. 对于希望尝试其他数据处理方法的研究人员，可直接使用原始采集数据。

## 关于本数据集
CEAP-360VR数据集由北京理工大学数字表演与仿真技术重点实验室与荷兰数学与计算机科学研究中心（Centrum Wiskunde & Informatica, CWI）分布式交互系统（Distributed & Interactive Systems, DIS）研究团队维护。
若需联系作者，请通过以下邮箱：
- 薛彤（Tong Xue）：[e-mail], [e-mail]
- Abdallah El Ali：[e-mail]

搜集汇总

背景与挑战

背景概述

CEAP-360VR是一个专注于360度VR视频情感计算的连续标注数据集，包含问卷、连续情感标注、行为数据（如头部和眼部运动）和生理数据（如心率和皮肤温度）。该数据集结构完整，提供原始和处理后的数据，适用于情感计算研究，并采用非商业许可。