EMOSIC: A Multimodal Musical Performance Dataset for Exploring the Emotional Synergy between Music and Motion

EMOSIC: Emotional Motion and Music Dataset Overview The **EMOSIC Dataset** features over **7.5 hours** of high-precision, synchronized audio and upper-body motion capture data. It captures the performances of **10 professional musicians** from a top-tier conservatory playing **traditional Chinese plucked instruments** (Zhong Ruan, Liuqin, Soprano Ruan, and Bass Ruan). This dataset provides a valuable **non-Eurocentric resource** to the affective computing and human-robot interaction communities. It is specifically designed to facilitate research into: - The **"Dominant-Amplifier"** mechanism in multimodal emotion perception. - Cross-cultural generalizability of emotion recognition models. - Generative motion synthesis for naturalistic robotic behaviors. Key Features - 90 Hz optical motion capture (15-marker upper-body skeletal model). - 16-bit, 16.0 kHz synchronized recordings. - Focuses on non-Western musical instrumentation and expressive gestures. --- Dataset Organization The dataset is organized into 3 main directories: ### 1. Data The complete dataset containing all **3,388 unique 8-second clips**. * **bvh/**: Processed 3D motion capture data in Biovision Hierarchy (`.bvh`) format. * **wav/**: Corresponding audio recordings in Waveform Audio File (`.wav`) format (16-bit, 16.0 kHz, mono). ### 2. Stimuli The subset of **60 clips** used as stimuli in the perceptual validation experiment described in the associated paper. These are partitioned into three modality conditions and three counterbalanced sets. * **A/**: Audio-Only clips (`.wav`). * **V/**: Visual-Only clips, rendered as silent skeletal animation videos (`.mp4`). * **AV/**: Audio-Visual clips, with synchronized audio and skeletal animations (`.mp4`). ### 3. Validation Contains raw data and metadata from the perceptual user study ($N=30$), provided in tabular (`.csv`) format. * **annotations_A.csv**: Raw Valence-Arousal ratings for Audio-Only condition. * **annotations_V.csv**: Raw Valence-Arousal ratings for Visual-Only condition. * **annotations_AV.csv**: Raw Valence-Arousal ratings for Audio-Visual condition. * **demographics.csv**: Participant demographic info (age, gender, musical background), linked by unique ID. * **participants.csv**: Counterbalanced group assignment for each participant. * **post_survey.csv**: Post-experiment qualitative feedback. --- Dataset Structure Once unzipped, the directory structure is as follows: ```text EmosicDataset_v1.0/ ├── README.md ├── Data/ │ ├── bvh/ # 3,388 .bvh files │ │ ├── 0000.bvh │ │ └── ... │ └── wav/ # 3,388 .wav files │ ├── 0000.wav │ └── ... ├── Stimuli/ # 60 clips used in user study │ ├── A/ │ │ ├── set1/ │ │ ├── set2/ │ │ └── set3/ │ ├── AV/ ... │ └── V/ ... └── Validation/ # User study results ├── annotations_A.csv ├── ... └── post_survey.csv ``` File Naming Convention The core data files in `Data/bvh` and `Data/wav` use a consistent 4-digit numerical identifier (from `0000` to `3387`). * **Example:** `0000.bvh` is the motion data corresponding strictly to `0000.wav`. Files within the `Stimuli` folders utilize the same identifiers, allowing easy mapping back to the full dataset. Citation If you use this dataset in your research, please cite the following paper: > **Asymmetrical Emotional Synergy between Music and Body Motion: Insights from the EMOSIC Musical Performance Dataset** > *Authors: Kaiyuan Ma, Biyun Zhang, Jun He* > *Scientific Reports, Under review* Contact For questions regarding the dataset, please contact: **Jun He** ([e-mail]) School of Mechanical Engineering, Shanghai Jiao Tong University