Research map / Reinforcement learning / Policy gradient and actor-critic

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Symmetric Q-learning, which adds learned GMM noise (fit to the negative Bellman error) to TD targets so the error distribution becomes symmetric, and applies it to SAC and REDQ on MuJoCo.

Problem

Least-squares value function training implicitly assumes Gaussian errors, but Bellman errors in online RL are often skewed, which can hurt learning efficiency.

Approach

Add zero-mean noise to the TD target, with the noise distribution modeled as a variational-Bayes GMM fit to the negative Bellman error, so the corrected error becomes approximately symmetric and better matches the least-squares assumption, without introducing bias in expectation.

Key results

• SymREDQ reached Humanoid return 5000 with 172K samples versus 255K for REDQ, and Walker2d 3500 with 77K versus 141K.
• At UTD=1 without ensemble, SymSAC matched SAC on Hopper, HalfCheetah and Ant and outperformed SAC on Walker2d and Humanoid.
• Visualizations show the GMM noise closely matches the negative Bellman error and the corrected error distribution is more symmetric than the pre-corrected one.

Measured effects reported in the paper

Surprising findings

• Extreme Q-learning variants (X-SAC, X-REDQ) performed poorly on most tasks, learning only on Hopper and HalfCheetah under the tested settings.
• Gains were mainly in learning speed rather than final return, and SymSAC without ensemble showed little improvement over SAC.

Limitations

• Added noise increases Q-value variance, so the method relies on a larger ensemble (N=20) to compensate, which raises compute cost.
• Performance is sensitive to the GMM update interval k, which must be tuned per task.
• Gains were mostly in sample efficiency; final return improvements were small or absent, and SymSAC without ensemble showed little gain.

Experimental setup

Datasets: MuJoCo Humanoid-v2 MuJoCo Walker2d-v2 MuJoCo Hopper-v2 MuJoCo Ant-v2 MuJoCo HalfCheetah-v2

Benchmarks: MuJoCo Humanoid-v2 MuJoCo Walker2d-v2 MuJoCo Hopper-v2 MuJoCo Ant-v2 MuJoCo HalfCheetah-v2

Metrics: average return sample efficiency (data needed to reach a return level) skewness of Bellman error distribution

Baselines: SAC REDQ X-SAC X-REDQ

Builds on: Soft Actor-Critic REDQ Extreme Q-learning Gumbel regression

Closest papers by meaning

• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.86)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.86)
• Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.86)
• Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics (ICLR 2025 · similarity 0.85)
• Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.85)