AFFEC Multimodal Dataset

Dataset: AFFEC - Advancing Face-to-Face Emotion Communication Dataset

Overview

The AFFEC (Advancing Face-to-Face Emotion Communication) dataset is a multimodal dataset designed for emotion recognition research. It captures dynamic human interactions through electroencephalography (EEG), eye-tracking, galvanic skin response (GSR), facial movements, and self-annotations, enabling the study of felt and perceived emotions in real-world face-to-face interactions. The dataset comprises 84 simulated emotional dialogues, 72 participants, and over 5,000 trials, annotated with more than 20,000 emotion labels.

Dataset Structure

The dataset follows the Brain Imaging Data Structure (BIDS) format and consists of the following components:

Root Folder:

sub-* : Individual subject folders (e.g., sub-aerj, sub-mdl, sub-xx2)

dataset_description.json: General dataset metadata

participants.json and participants.tsv: Participant demographics and attributes

task-fer_events.json: Event annotations for the FER task

README.md: This documentation file

Subject Folders (sub-):

Each subject folder contains:

Behavioral Data (beh/): Physiological recordings (eye tracking, GSR, facial analysis, cursor tracking) in JSON and TSV formats.

EEG Data (eeg/): EEG recordings in .edf and corresponding metadata in .json.

Event Files (*.tsv): Trial event data for the emotion recognition task.

Channel Descriptions (*_channels.tsv): EEG channel information.

Data Modalities and Channels

1. Eye Tracking Data

Channels: 16 (fixation points, left/right eye gaze coordinates, gaze validity)

Sampling Rate: 62 Hz

Trials: 5632

File Example: sub-_task-fer_run-0_recording-gaze_physio.json

2. Pupil Data

Channels: 21 (pupil diameter, eye position, pupil validity flags)

Sampling Rate: 149 Hz

Trials: 5632

File Example: sub-_task-fer_run-0_recording-pupil_physio.json

3. Cursor Tracking Data

Channels: 4 (cursor X, cursor Y, cursor state)

Sampling Rate: 62 Hz

Trials: 5632

File Example: sub-_task-fer_run-0_recording-cursor_physio.json

4. Face Analysis Data

Channels: Over 200 (2D/3D facial landmarks, gaze detection, facial action units)

Sampling Rate: 40 Hz

Trials: 5680

File Example: sub-_task-fer_run-0_recording-videostream_physio.json

5. Electrodermal Activity (EDA) and Physiological Sensors

Channels: 40 (GSR, body temperature, accelerometer data)

Sampling Rate: 50 Hz

Trials: 5438

File Example: sub-_task-fer_run-0_recording-gsr_physio.json

6. EEG Data

Channels: 63 (EEG electrodes following the 10-20 placement scheme)

Sampling Rate: 256 Hz

Reference: Left earlobe

Trials: 5632

File Example: sub-_task-fer_run-0_eeg.edf

7. Self-Annotations

Trials: 5807

Annotations Per Trial: 4

Event Markers: Onset time, duration, trial type, emotion labels

File Example: task-fer_events.json

Experimental Setup

Participants engaged in a Facial Emotion Recognition (FER) task, where they watched emotionally expressive video stimuli while their physiological and behavioral responses were recorded. Participants provided self-reported ratings for both perceived and felt emotions, differentiating between the emotions they believed the video conveyed and their internal affective experience.

The dataset enables the study of individual differences in emotional perception and expression by incorporating Big Five personality trait assessments and demographic variables.

Usage Notes

The dataset is formatted in ASCII/UTF-8 encoding.

Each modality is stored in JSON, TSV, or EDF format as per BIDS standards.

Researchers should cite this dataset appropriately in publications.

Applications

AFFEC is well-suited for research in:

Affective Computing

Human-Agent Interaction

Emotion Recognition and Classification

Multimodal Signal Processing

Neuroscience and Cognitive Modeling

Healthcare and Mental Health Monitoring

Acknowledgments

This dataset was collected with the support of brAIn lab, IT University of Copenhagen. Special thanks to all participants and research staff involved in data collection.

License

This dataset is shared under the Creative Commons CC0 License.

Contact

For questions or collaboration inquiries, please contact [[e-mail]].