DIPSEER: A Dataset for In-Person Student Emotion and Engagement Recognition in the Wild

Data DescriptionThe DIPSER dataset is designed to assess student attention and emotion in in-person classroom settings, consisting of RGB camera data, smartwatch sensor data, and labeled attention and emotion metrics. It includes multiple camera angles per student to capture posture and facial expressions, complemented by smartwatch data for inertial and biometric metrics. Attention and emotion labels are derived from self-reports and expert evaluations. The dataset includes diverse demographic groups, with data collected in real-world classroom environments, facilitating the training of machine learning models for predicting attention and correlating it with emotional states.Data Collection and Generation ProceduresThe dataset was collected in a natural classroom environment at the University of Alicante, Spain. The recording setup consisted of six general cameras positioned to capture the overall classroom context and individual cameras placed at each student’s desk. Additionally, smartwatches were used to collect biometric data, such as heart rate, accelerometer, and gyroscope readings.Experimental SessionsNine distinct educational activities were designed to ensure a comprehensive range of engagement scenarios:News Reading – Students read projected or device-displayed news.Brainstorming Session – Idea generation for problem-solving.Lecture – Passive listening to an instructor-led session.Information Organization – Synthesizing information from different sources.Lecture Test – Assessment of lecture content via mobile devices.Individual Presentations – Students present their projects.Knowledge Test – Conducted using Kahoot.Robotics Experimentation – Hands-on session with robotics.MTINY Activity Design – Development of educational activities with computational thinking.Technical SpecificationsRGB Cameras: Individual cameras recorded at 640×480 pixels, while context cameras captured at 1280×720 pixels.Frame Rate: 9-10 FPS depending on the setup.Smartwatch Sensors: Collected heart rate, accelerometer, gyroscope, rotation vector, and light sensor data at a frequency of 1–100 Hz.Data Organization and FormatsThe dataset follows a structured directory format:/groupX/experimentY/subjectZ.zip
Each subject-specific folder contains:images/ (individual facial images)watch_sensors/ (sensor readings in JSON format)labels/ (engagement & emotion annotations)metadata/ (subject demographics & session details)Annotations and LabelingEach data entry includes engagement levels (1-5) and emotional states (9 categories) based on both self-reported labels and evaluations by four independent experts. A custom annotation tool was developed to ensure consistency across evaluations.Missing Data and Data QualitySynchronization: A centralized server ensured time alignment across devices. Brightness changes were used to verify synchronization.Completeness: No major missing data, except for occasional random frame drops due to embedded device performance.Data Consistency: Uniform collection methodology across sessions, ensuring high reliability.Data Processing MethodsTo enhance usability, the dataset includes preprocessed bounding boxes for face, body, and hands, along with gaze estimation and head pose annotations. These were generated using YOLO, MediaPipe, and DeepFace.File Formats and AccessibilityImages: Stored in standard JPEG format.Sensor Data: Provided as structured JSON files.Labels: Available as CSV files with timestamps.The dataset is publicly available under the CC-BY license and can be accessed along with the necessary processing scripts via the DIPSER GitHub repository.Potential Errors and LimitationsDue to camera angles, some student movements may be out of frame in collaborative sessions.Lighting conditions vary slightly across experiments.Sensor latency variations are minimal but exist due to embedded device constraints.CitationIf you find this project helpful for your research, please cite our work using the following bibtex entry:@misc{marquezcarpintero2025dipserdatasetinpersonstudent1,
   title={DIPSER: A Dataset for In-Person Student Engagement Recognition in the Wild}, 
   author={Luis Marquez-Carpintero and Sergio Suescun-Ferrandiz and Carolina Lorenzo Álvarez and Jorge Fernandez-Herrero and Diego Viejo and Rosabel Roig-Vila and Miguel Cazorla},
   year={2025},
   eprint={2502.20209},
   archivePrefix={arXiv},
   primaryClass={cs.CV},
   url={https://arxiv.org/abs/2502.20209}, 
}
Usage and ReproducibilityResearchers can utilize standard tools like OpenCV, TensorFlow, and PyTorch for analysis. The dataset supports research in machine learning, affective computing, and education analytics, offering a unique resource for engagement and attention studies in real-world classroom environments.

### 数据描述
DIPSER数据集旨在评估线下课堂场景中学生的注意力与情感状态，包含RGB摄像头数据、智能手表传感器数据以及标注后的注意力与情感指标。本数据集为每位学生设置多机位以捕捉其姿势与面部表情，并辅以智能手表采集的惯性与生物特征指标数据。注意力与情感标签来源于自我报告与专家评估。数据集涵盖多样化的人口统计学群体，所有数据均采集自真实课堂环境，可用于训练预测注意力并将其与情感状态相关联的机器学习模型。

### 数据采集与生成流程
本数据集采集自西班牙阿利坎特大学的自然课堂环境。录制部署包含6台通用摄像头以捕捉整体课堂场景，同时在每位学生的课桌旁部署独立摄像头。此外，研究团队使用智能手表采集心率、加速度计与陀螺仪读数等生物特征数据。

### 实验会话
本次研究设计了9种不同的教学活动，以覆盖全面的课堂参与场景：
1. 新闻阅读（News Reading）：学生阅读投影或设备展示的新闻内容
2. 头脑风暴活动（Brainstorming Session）：围绕问题解决开展创意构思
3. 授课环节（Lecture）：被动聆听讲师主导的课程
4. 信息整理（Information Organization）：整合不同来源的信息
5. 授课内容测验（Lecture Test）：通过移动设备完成授课内容测评
6. 个人展示环节（Individual Presentations）：学生展示其项目成果
7. 知识测验（Knowledge Test）：采用Kahoot平台开展的知识测评
8. 机器人实验实践（Robotics Experimentation）：动手操作机器人的实践课程
9. MTINY教学活动设计（MTINY Activity Design）：结合计算思维开展教学活动开发

### 技术规格
RGB摄像头：独立摄像头的录制分辨率为640×480像素，场景摄像头的录制分辨率为1280×720像素。
帧率：根据部署配置不同，为9-10 FPS。
智能手表传感器：以1–100 Hz的频率采集心率、加速度计、陀螺仪、旋转矢量以及光线传感器数据。

### 数据组织与格式
数据集采用结构化目录格式：/groupX/experimentY/subjectZ.zip
每个受试者专属文件夹包含以下内容：
- images/：面部特写图像
- watch_sensors/：JSON格式的传感器读数文件
- labels/：参与度与情感标注文件
- metadata/：受试者人口统计学信息与实验会话详情

### 标注与标记流程
每条数据包含1-5级的参与度评分与9类情感状态标签，均基于自我报告与四位独立专家的评估结果。研究团队开发了自定义标注工具以确保不同评估者间的标注一致性。

### 缺失数据与数据质量
同步性：通过中心化服务器实现多设备时间对齐，并使用亮度变化验证同步效果。
完整性：无大规模数据缺失，仅偶发因嵌入式设备性能问题导致的帧丢失。
数据一致性：所有实验会话采用统一的采集流程，保障数据的高可靠性。

### 数据处理方法
为提升数据集的易用性，本数据集包含人脸、身体与手部的预处理边界框，以及视线估计与头部姿态标注内容，这些内容通过YOLO、MediaPipe与DeepFace生成。

### 文件格式与可访问性
图像：采用标准JPEG格式存储。
传感器数据：以结构化JSON文件格式提供。
标签：为带有时间戳的CSV格式文件。
本数据集采用CC-BY许可公开，可通过DIPSER的GitHub仓库获取数据集与配套的处理脚本。

### 潜在误差与局限性
受拍摄视角限制，在协作场景中部分学生的动作可能超出画面范围。
不同实验环节的光照条件存在细微差异。
受嵌入式设备约束，传感器存在微小的延迟偏差。

### 引用方式
若本数据集对您的研究有所帮助，请使用以下BibTeX条目进行引用：
@misc{marquezcarpintero2025dipserdatasetinpersonstudent1,
title={DIPSER: A Dataset for In-Person Student Engagement Recognition in the Wild},
author={Luis Marquez-Carpintero and Sergio Suescun-Ferrandiz and Carolina Lorenzo Álvarez and Jorge Fernandez-Herrero and Diego Viejo and Rosabel Roig-Vila and Miguel Cazorla},
year={2025},
eprint={2502.20209},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2502.20209},
}

### 使用与可复现性
研究人员可使用OpenCV、TensorFlow、PyTorch等标准工具开展数据分析。本数据集支持机器学习、情感计算与教育分析领域的研究，为真实课堂环境中的参与度与注意力研究提供了独特的优质资源。

搜集汇总

背景与挑战

背景概述

DIPSEER数据集是一个用于识别真实课堂中学生情绪和参与度的多模态数据集，包含RGB摄像头数据、智能手表传感器数据和标注的注意力及情绪指标。数据采集于自然课堂环境，涵盖多种教育活动场景，结构清晰且经过预处理，适用于机器学习和情感计算研究。