Delay-Aware Lyapunov-Enhanced Multi-Agent Soft Actor-Critic Algorithm

Ensuring stability in cooperative multi-agent systems remains a principal challenge for multi-agent reinforcement learning, particularly when agents operate over communication networks with delays and external disturbances. This paper proposes the Lyapunov-enhanced Soft Actor-Critic Algorithm with Delay Compensation (LSAC-delay), which integrates Lyapunov stability theory and algebraic graph theory into the maximum-entropy Multi-Agent Reinforcement Learning (MARL) framework. The algorithm defines a consensus-based error over a communication graph and employs delay-aware Lyapunov penalties that adaptively scale based on current communication conditions, enabling fully decentralized execution while maintaining provable stability guarantees. We establish exponential stability in mean square for the ideal case without delays or disturbances. Extending this result, we prove exponential ultimate boundedness for the realistic case with bounded communication delays and external disturbances, providing explicit bounds on the system's ultimate error in terms of delay and disturbance magnitudes. The experimental results show that, compared with the baseline algorithm, LSAC-delay has better performance and robustness in the delay scenario, verifying its effectiveness in secure and reliable cooperative control in the network multi-agent system.