A constrained reinforcement learning based approach for cooperative control of multi-UAV in dense obstacle environments

Formation control of multiple unmanned aerial vehicles (UAVs) is a fundamental challenge for advanced cooperative tasks. In dense obstacle environments, machine learning-based formation control algorithms face significant challenges due to the high environmental complexity and UAV dynamics, particularly manifested as an explosion in state space dimensionality and poor obstacle avoidance robustness. To address these issues, this paper proposes a threat-aware subspace feature extraction mechanism and constraint learning algorithm within a deep reinforcement learning (DRL) framework. Our approach first reconstructs spatial distributions of high-threat obstacles and UAV kinematic characteristics from LiDAR point cloud data to reduce state space dimensionality. Then, we solve the obstacle avoidance problem using a novel constrained reinforcement learning framework. This framework employs a safety-oriented penalty function instead of conventional posterior penalties to explicitly enforce safety constraints, thereby preventing dangerous actions. We rigorously prove the algorithm's convergence and stability using Lyapunov stability theory. Comparative experiments carried out in the high-fidelity AirSim environment have demonstrated that the proposed algorithm outperforms the state-of-the-art methods, where the convergence speed improves by $36.36~%$, stability increases by $18.22~%$, and mission success rate rises by $5.6~%$.