Convergence test of projected momentum gradient descent for varying computing and transmission resouces

In this work, we address the latency issue in federated deep reinforcement learning using Lyapunov Drift plus Penalty (LDDP). The LDDP effectively eliminates queues in the system, ensuring that the penalty term allocates resources efficiently to users during each federation round. To make the penalty term dynamic, we employ Projected Momentum Gradient Descent (PMGD) to adaptively adjust the parameters in the penalty term, such as the Lyapunov tuning parameter, the weight parameter (which serves as a trade-off balance), and the computational and transmission resources. Extensive simulations demonstrate that PMGD outperforms the Alternating Direction Method of Multipliers (ADMM) in terms of convergence speed, and PMGD can dynamically adjust penalty parameters.