Adaptive optimal control of discrete-time systems by reinforcement learning: damping coefficients-based stabilizing policy iterations

Policy iteration is one of the classical frameworks of adaptive dynamic programming, which requires a known initial stabilizing control to start the iteration. To relax this requirement, two different stabilizing policy iteration algorithms based on variable damping coefficients are designed for unknown discrete-time linear systems. First, we design a stabilizing artificial system and then iterate it gradually to the original system by cumulating damping coefficients and thus obtaining a stabilizing control policy. Then, a data-driven version of the stabilizing policy iteration framework is designed, and the corresponding model-free scheme is proposed for determining the damping coefficients. To relax the same initial condition that exists in traditional policy iteration-based $\mathcal{Q}$-learning, another novel data-driven $\mathcal{Q}$-learning algorithm based on stabilizing policy iteration is developed. The proposed $\mathcal{Q}$-learning algorithm is equivalent to the stabilizing policy iteration framework by theoretical analysis. Ultimately, the effectiveness of the two proposed algorithms is verified by a numerical example.