HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

HARMES is a multi-modal dataset for human activity recognition (HAR), comprised of 20 participants conducting 15 activities of daily living (ADLs). 61 hours of it are fully labeled, and we include 1 additional hour of other, free-form, mostly labeled activities on top (total: 20h). We make available two versions: HARMES-RAW.zip: Full dataset upload. Contains all recorded raw sensor data, no resampling, no scaling. Data is already synchronized with timestamps in the "ts_sync" column. HARMES-preprocessed.pkl: Python pickle files (one per participant), containing lists of recordings. Recordings were resampled to 50 Hz, and contain per-recording-z-score-normalized IMU sensor values, BME280 atmospheric sensor values, and the audio data. The preprocessed data can be loaded like this (iterate over participant-IDs to load all data): import pickle with open("participant_01.pkl", "rb") as f: data = pickle.load(f) The code for loading, processing, and using the raw data in deep learning experiments is available on our GitHub page (see below). For each participant, we supply four recordings of about 1h each. The first three recordings are fully labeled, while recording xx04 is always the free-form recording. Some of the recordings are split into two parts, where part two is a direct continuation of part one. Structure of the raw data: Dataset Participant Folders (01-20) Recording Folders (xy01-xy04), where xy is the participant ID <timestamp1>_TASKLIST.csv: contains the order of tasks that the participant performed (Note: these are included, but unused for 4th recordings) <timestamp2>.csv: contains all activity labels and annotator's comments for special cases <rec_id>_merged.csv: contains all data recorded from the Puck.js - 6-axis IMU (left wrist), 3 axis BME280. recording_<date_str>.csv: contains IMU data recorded from the smartwatch - 6-axis IMU (right wrist). recording_<date_str>_SYNC_H5.h5: HDF5-File containing the microphone recording data (44.1kHz), from the right wrist. Structure of the preprocessed data: Dataset (stored as one Python pickle file per participant): Filename: participant_<Participant ID>.pkl Content: List containing the participant's recordings as dictionaries. Each participant has four/five recordings Each recording is a dictionary with the following keys: IMU_L (left IMU data, 50Hz) IMU_R (right IMU data, 50Hz) Audio (audio signal, 44.1kHz) BME280 (environmental sensor data, 50Hz) Labels (Annotations with start/stop times in ms) rec_id (ID of the recording, same as the folder name of the recording in the raw version.) If you use this data, please cite our paper and this repository: (citation will be added soon)