Air-road cross domain collaborative hierarchical reinforcement learning for real-time formation control: A network signal generator approach

In this paper, a hierarchical reinforcement learning (HRL) based real-time formation control approach is proposed for heterogeneous aerial-ground agents (HAGAs).Initially, to address the issue of imprecise modeling of HAGAs, a unified heterogeneous chained system model is constructed using the hand-position method.Subsequently, a hierarchical framework is designed:(1) To decouple multi-agent collaborative interactions and individual dynamic rules through hierarchical resolution, which enables controller design to be independent of direct reliance on neighborhood collaborative errors.(2) By adopting a dual-layer framework that separates collaborative topology management from individual control strategies, seamless switching between multiple task scenarios can be achieved simply by reconstructing the collaborative topology of the first layer.Moreover, to overcome the issue of non-asymptotic stability of tracking errors caused by the discount factor in traditional optimal control, a cost function based on the derivative of the tracking error is introduced.This not only addresses the error issue caused by the discount factor but also effectively resolves the problem of the unboundedness of the quadratic cost function.Finally, the efficacy of the proposed algorithm is substantiated through simulation experiments.