Enhancing off-policy optimisation in structured Markov decision processes via Thompson Sampling

Reinforcement Learning (RL) provides a framework for solving sequential decision-making tasks, yet limited and potentially unsafe data hinder its training process. Off-policy algorithms are commonly employed to mitigate these issues, but they face challenges in data-scarce or non-ergodic environments and often exhibit exploding variance over long trajectories. We propose a novel algorithm that integrates Thompson Sampling, originally developed for multi-armed bandit problems, to enable efficient and safe policy identification. By exploiting the structural properties of Structured Markov Decision Processes (SMDPs), our approach reduces the policy search space, enhances learning stability, and demonstrates superior performance compared to Q-learning, SOCU, and SOCU-v2.