mRI: multi-modal 3d human pose estimation dataset using mmwave, rgb-d, and inertial sensors

The ability to estimate 3D human body pose and movement, also known as human pose estimation~(HPE), enables many applications for home-based health monitoring, such as remote rehabilitation training. Several possible solutions have emerged using sensors ranging from RGB cameras, depth sensors, millimeter-Wave (mmWave) radars, and wearable inertial sensors. Despite previous efforts on datasets and benchmarks for HPE, few datasets exploit multiple modalities and focus on home-based health monitoring.
To bridge this gap, we present mRI, a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors. Our dataset consists of over 5 million frames from 20 subjects performing rehabilitation exercises and supports the benchmarks of HPE and action detection. We perform extensive experiments using our dataset and delineate the strength of each modality.
We hope that the release of mRI can catalyze the research in pose estimation, multi-modal learning, and action understan...

人体3D姿态与运动估计技术，又称人体姿态估计（human pose estimation, HPE），能够赋能诸多居家健康监测类应用，例如远程康复训练。目前已涌现出多种基于不同传感器的解决方案，涵盖RGB相机、深度传感器、毫米波（millimeter-Wave, mmWave）雷达以及可穿戴惯性传感器等类型。尽管此前已有诸多针对HPE的数据集与基准测试集相关研究，但鲜有数据集能够兼顾多模态特性且聚焦于居家健康监测场景。为填补这一研究空白，本文提出mRI：一款集成毫米波、RGB-D与惯性传感器的多模态3D人体姿态估计数据集。该数据集包含20名受试者完成康复训练动作的超500万帧数据，可支撑人体姿态估计与动作检测相关的基准测试任务。我们基于该数据集开展了大量实验，并详细阐明了各模态的性能优势。我们期望mRI数据集的发布能够推动姿态估计、多模态学习以及动作理解等领域的研究进展……