A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT

In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.

KEYWORDS:

• Markov switched interconnected systems
• reinforcement learning
• optimal control
• homotopy-based policy iteration

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work is supported by the National Natural Science Foundation of China [grant numbers 62103005, 62173001 and 62273006]. The Natural Science Foundation for Distinguished Young Scholars of Higher Education Institutions of Anhui Province [grant number 2022AH020034], the Natural Science Foundation for Excellent Young Scholars of Higher Education Institutions of Anhui Province [grant numbers 2023AH030030 and 2022AH030049], the Research and development project of Engineering Research Center of Biofilm Water Purification and Utilization Technology of Ministry of Education [grant number BWPU2023ZY02] and the University Synergy Innovation Program of Anhui Province [grant number GXXT-2023-020].

Notes on contributors

Jinxu Liu

Jinxu Liu is now a M.S. candidate at the School of Electrical and Information Engineering, Anhui University of Technology, China. His current research interests include interconnected systems, Markov jump systems, reinforcement learning and optimal control.

Xuanrui Mi

Xuanrui Mi received the B.S. degree in automation in 2021 from the Xian University of Science and Technology, Xi'an, China. He is currently pursuing the M.S. degree in control engineering with School of Electrical and Information Engineering, Anhui University of Technology, Ma'anshan. His current research interests include interconnected systems, reinforcement learning and singularly perturbed systems.

Jianwei Xia

Jianwei Xia received the M.S. degree in automatic engineering from Qufu Normal University, Qufu, China in 2004 and the Ph.D. degree in automatic control from the Nanjing University of Science and Technology, Nanjing, China in 2007. He is a Professor with the School of Mathematics Science, Liaocheng University, Liaocheng, China. From 2010 to 2012, he was a Postdoctoral Research Associate with the School of Automation, Southeast University, Nanjing. From 2013 to 2014, he was a Postdoctoral Research

Lei Su

Lei Su received the M.S. degree in School of Electrical and Information Engineering, Anhui University of Technology, Ma'anshan, China, in 2016 and the Ph.D. degree in control theory and engineering from the Northeastern University in 2020. Now, he is a associate professor of the School of Electrical and Information Engineering, Anhui University of Technology, China. His research interests include fault-tolerant control, event-triggered control, Markov jump systems and cyber-physical systems.

Hao Shen

Hao Shen received the Ph.D. degree in control theory and control engineering from Nanjing University of Science and Technology, Nanjing, China, in 2011. Since 2011, he has been with Anhui University of Technology, China, where he is currently a Professor. His current research interests include stochastic hybrid systems, complex networks, fuzzy systems and control, nonlinear control. Dr. Shen has served on the technical program committee for several international conferences. He is an Associate Editor/Guest Editor for several international journals, including Journal of The Franklin Institute, Applied Mathematics and Computation, Neural Processing Letters and Transactions of the Institute Measurement and Control. Prof. Shen was a recipient of the Highly Cited Researcher Award by Clarivate Analytics (formerly, Thomson Reuters) in 2019-2021.