Virtual IR Sensing for Planetary Rovers: Improved Terrain Classification and Thermal Inertia Estimation

Abstract—Terrain classification is critically important forMars rovers, which rely on it for planning and autonomous navigation.On-board terrain classification using visual informationhas limitations, and is sensitive to illumination conditions.Classification can be improved if one fuses visual imagery withadditional infrared (IR) imagery of the scene, yet unfortunatelythere are no IR image sensors on the current Mars rovers. Avirtual IR sensor, estimating IR from RGB imagery using deeplearning, was proposed in the context of a MU-Net architecture.However, virtual IR estimation was limited by the fact thatslope angle variations induce temperature differences within thesame terrain. This paper removes this limitation, giving good IRestimates and as a consequence improving terrain classificationby including the additional angle from the surface normal to theSun and the measurement of solar radiation. The estimates arealso useful when estimating thermal inertia, which can enhanceslip prediction and small rock density estimation. Our approachis demonstrated in two applications. We collected a new dataset to verify the effectiveness of the proposed approach andshow its benefit by applying to the two applications.