mRI: multi-modal 3d human pose estimation dataset using mmwave, rgb-d, and inertial sensors

The ability to estimate 3D human body pose and movement, also known as
human pose estimation~(HPE), enables many applications for home-based
health monitoring, such as remote rehabilitation training. Several
possible solutions have emerged using sensors ranging from RGB cameras,
depth sensors, millimeter-Wave (mmWave) radars, and wearable inertial
sensors. Despite previous efforts on datasets and benchmarks for HPE, few
datasets exploit multiple modalities and focus on home-based health
monitoring. To bridge this gap, we present mRI, a multi-modal 3D
human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors.
Our dataset consists of over 5 million frames from 20 subjects performing
rehabilitation exercises and supports the benchmarks of HPE and action
detection. We perform extensive experiments using our dataset and
delineate the strength of each modality. We hope that the release
of mRI can catalyze the research in pose estimation, multi-modal
learning, and action understanding, and more importantly, facilitate the
applications of home-based health monitoring.