Wearable RGB Camera Images of Human Locomotion Environments

Drawing inspiration from autonomous vehicles, using future environment information could improve the control of wearable biomechatronic devices for assisting human locomotion. To the authors knowledge, this research represents the first documented investigation using machine vision and deep convolutional neural networks for environment recognition to support the predictive control of robotic lower-limb prostheses and exoskeletons. One participant was instrumented with a battery-powered, chest-mounted RGB camera system. Approximately 10 hours of video footage were experimentally collected while ambulating throughout unknown outdoor and indoor environments. The sampled images were preprocessed and individually labelled. A deep convolutional neural network was developed and trained to automatically recognize three walking environments: level-ground, incline staircases, and decline staircases. The environment recognition system achieved 94.85% overall image classification accuracy. Extending these preliminary findings, future research should incorporate other environment classes (e.g., incline ramps) and integrate the environment recognition system with electromechanical sensors and/or surface electromyography for automated locomotion mode recognition. The challenges associated with implementing deep learning on wearable biomechatronic devices are discussed.Reference: Laschowski B, McNally W, McPhee J, and Wong A. (2019). Preliminary Design of an Environment Recognition System for Controlling Robotic Lower-Limb Prostheses and Exoskeletons. IEEE International Conference on Rehabilitation Robotics (ICORR), pp. 868-873. DOI: 10.1109/ICORR.2019.8779540.本研究汲取自动驾驶汽车的研发思路，提出引入未来环境信息可优化用于辅助人类行走的可穿戴生物机电设备（wearable biomechatronic devices）的控制策略。据作者所知，本研究首次采用机器视觉（machine vision）与深度卷积神经网络（deep convolutional neural networks）开展环境识别研究，以支持机器人下肢假肢（robotic lower-limb prostheses）与外骨骼（exoskeletons）的预测控制。 一名受试者配备了电池供电、安装于胸部的RGB摄像系统（RGB camera system）。研究人员在未知的室外与室内环境中步行时，实验采集了约10小时的视频素材。所采集的图像经预处理后完成单帧标注。本研究开发并训练了一款深度卷积神经网络，可自动识别三类行走环境：平地、上行楼梯与下行楼梯。该环境识别系统的整体图像分类准确率达94.85%。 基于上述初步研究成果，未来研究可拓展更多环境类别（例如斜坡），并将环境识别系统与机电传感器（electromechanical sensors）和/或表面肌电图（surface electromyography）相结合，以实现自动化行走模式识别。本文还探讨了在可穿戴生物机电设备上部署深度学习所面临的相关挑战。 参考文献：Laschowski B、McNally W、McPhee J 与 Wong A.（2019）. 用于控制机器人下肢假肢与外骨骼的环境识别系统初步设计. 收录于IEEE国际康复机器人学会议（ICORR）论文集，第868至873页. DOI: 10.1109/ICORR.2019.8779540.