2 / 2025-03-04 14:44:57

Action Candidate-Driven Clipped Double Q-Learning in Continuous Control Tasks

Deep reinforcement learning, Clipped double Q-learning, Action candidate, Estimation bias

    Deep reinforcement learning algorithms have been widely used in continuous control tasks. However, they frequently encounter challenges in accurately estimating Q-functions and suffer performance degradation in high-dimensional state-action spaces. To alleviate these problems, we propose TD6 (TD3 + 3 additions), a novel algorithm extending the twin delayed deep deterministic policy gradient (TD3) algorithm. TD6 introduces three critical enhancements. A shared set of action candidates enables two Q-networks to select optimal action independently. Building upon double Q-learning principles, TD6 cross-evaluates optimal actions between the two Q-networks, separating action selection and evaluation to improve stability and estimation precision. TD6 integrates action candidates with clipped double Q-learning, moderated by a regularization factor, to balance the overestimation bias inherent in deep deterministic policy gradients and the underestimation bias observed in TD3. By leveraging these enhancements, TD6 achieves superior performance in complex environments. Experimental results demonstrate that TD6 significantly reduces estimation bias and outperforms TD3 and DDPG in continuous control benchmarks, showing its superiority.