Demonstration of the Dyna Reinforcement Learning Framework for Reactive Close Proximity Operations

Lessons from the International Space Station (ISS) emphasize the necessity of exterior inspection for anomaly detection and maintenance, but current methods rely on costly and limited human extravehicular activities and robotic arms. Deployable free-flying small spacecraft offer a flexible, autonomous solution, capable of comprehensive exterior inspections without human involvement. However, the safety of these spacecraft during close proximity operations remains a concern, particularly given the variability in thruster performance. This paper presents SmallSat Steward, a reactive and integrated architecture for online model learning and trajectory planning based on the Dyna reinforcement learning architecture. By combining model-based planning and direct reinforcement learning, Dyna offers a flexible and computationally-efficient solution capable of adapting to changes in thruster performance and other system uncertainties. Preliminary results in both simulation and hardware environments demonstrate the potential of this architecture to successfully regulate position under significant model degradation