Reinforcement Learning-Based Hybrid Force/Position Control of Redundant Manipulators under Time Delays

Hybrid force/position control of redundant robotic manipulators under time delays, disturbances, and model uncertainties remains a significant challenge. Conventional methods rely on separating the force and position control spaces and on precise system identification, which increases complexity and reduces robustness. This paper introduces a semi-model-free framework, the Force/Position Reinforcement Learning Super-Twisting Algorithm (F/P-RL-STA), which avoids explicit space separation and reduces dependency on accurate models. By integrating a Super-Twisting Sliding Mode Controller (STSMC) with Deep Deterministic Policy Gradient (DDPG) reinforcement learning, the proposed method enables simultaneous force and position regulation. The reinforcement learning agent adaptively tunes control parameters and null-space torques, while ensuring smooth transitions between free motion and contact interaction. This adaptive mechanism reduces control effort, suppresses chattering, and enhances robustness against disturbances and time delays. Closed-loop stability is guaranteed through Lyapunov analysis. Sensitivity analysis is conducted to guide the tuning of learning parameters and reward design. Simulation results confirm that the proposed approach significantly outperforms conventional methods in terms of accuracy, energy efficiency, smoothness of control signals, and chattering elimination, even in the presence of disturbances and time delays.An agent reinforcement learning architecture is introduced to switch smoothly between position control during free motion and hybrid force/position control during contact tasks. The kinematic redundancy of the OpenManipulator-X robot is exploited using a learned null-space torque to minimize control effort and suppress chattering. A comprehensive sensitivity analysis is performed to guide optimal tuning of learning parameters and the reward function, enhancing training stability and controller robustness. Simulation results confirm that F/P-RL-STA significantly outperforms both SMC-HFPC and F/P-STA in terms of tracking precision, control smoothness, disturbance rejection, and energy efficiency, while maintaining closed-loop stability as verified through Lyapunov-based analysis.

存在时滞、扰动与模型不确定性的冗余度机械臂混合力位控制，仍是一项亟待攻克的重大挑战。传统方法依赖于力控与位控空间的分离设计以及精确的系统辨识，这不仅提升了控制复杂度，还降低了系统鲁棒性。本文提出一种半无模型框架——力位强化学习超扭曲算法（Force/Position Reinforcement Learning Super-Twisting Algorithm，F/P-RL-STA），该框架无需显式划分控制空间，同时降低了对精确系统模型的依赖。通过将超扭曲滑模控制器（Super-Twisting Sliding Mode Controller，STSMC）与深度确定性策略梯度（Deep Deterministic Policy Gradient，DDPG）强化学习相结合，所提方法可同时实现力控与位控调节。该强化学习智能体可自适应调整控制参数与零空间力矩，同时保障自由运动与接触交互间的平滑过渡。该自适应机制能够降低控制能耗、抑制抖振，并提升系统对扰动与时滞的鲁棒性。通过李雅普诺夫分析可确保闭环系统的稳定性。本文开展了灵敏度分析，以指导学习参数与奖励函数的整定。仿真结果表明，即便存在扰动与时滞，所提方法在控制精度、能效、控制信号平滑性以及抖振抑制方面均显著优于传统方法。
本文提出一种强化学习智能体架构，可实现自由运动阶段的位控与接触任务阶段的混合力位控制之间的平滑切换。本文利用学习得到的零空间力矩，对OpenManipulator-X机械臂的运动学冗余特性进行优化，以降低控制能耗并抑制抖振。本文开展了全面的灵敏度分析，以指导学习参数与奖励函数的最优整定，进而提升训练稳定性与控制器鲁棒性。仿真结果验证，在通过李雅普诺夫分析确保闭环稳定性的前提下，F/P-RL-STA在跟踪精度、控制平滑性、抗扰性能与能效方面均显著优于SMC-HFPC与F/P-STA两种方法。