Research map / Reinforcement learning / Policy gradient and actor-critic

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Stabilizing deep actor-critic · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Annealed Q-learning, which anneals the expectile parameter tau from near 1 to 0.5 in the critic loss of TD3/SAC, moving from the Bellman optimality operator to the Bellman operator for faster, less biased online actor-critic learning in continuous control.

Problem

Actor-critic methods for continuous actions evaluate the current policy with the Bellman operator, which relies on policy updates for improvement and gives low sample efficiency; using the Bellman optimality operator speeds learning but causes overestimation bias.

Approach

Replace the L2 critic loss in TD3/SAC with an expectile loss whose tau is linearly annealed from a value near 1 (optimality-like, max-approximating, exploration-promoting) down to 0.5 (SARSA-like, mean estimation), so the operator transitions smoothly from Bellman optimality to Bellman without a second Q-function.

Key results

• On 10 DM Control tasks at 3M steps, AQ-TD3 reached mean 740.3 and AQ-SAC 746.1, versus TD3 492.7 and SAC 657.9.
• AQ-SAC with annealing from tau=0.9 (mean 746.1) outperformed fixed-tau variants (best fixed 0.7 at 730.7; fixed 0.95 at 364.9), and was robust to tau_init between 0.7 and 0.95.
• Annealed AQ-SAC improved sample efficiency: at 1M steps AQ-SAC scored 624.8 mean versus SAC 493.7.

Measured effects reported in the paper

Surprising findings

• Fixed expectile values above 0.8 severely degrade performance (fixed 0.95 mean 364.9), while annealing from high tau remains stable.
• Increasing overestimation bias early in training raised policy entropy, which the authors interpret as promoting exploration.

Limitations

• Annealing schedule is a single hyperparameter-driven linear decay; non-linear schedules (Exp2) performed worse and dynamic schedules were left for future work.
• Experiments are limited to simple state-based DM Control and Meta-World tasks in a single-task setting; the preliminary analysis uses a tabular toy MDP.
• Overestimation bias remains an open concern; combining with ensembles was not explored.

Experimental setup

Datasets: DM Control (10 locomotion tasks) Meta-World (10 manipulation tasks)

Benchmarks: DM Control Meta-World

Metrics: mean return interquartile mean (IQM) success rate confidence intervals Q-value bias relative to Monte Carlo return policy entropy

Baselines: TD3 SAC XQL (XSAC) max-backup MXQL Annealed TD3 comparison with fixed tau

Builds on: TD3 SAC Implicit Q-learning XQL MXQL

Closest papers by meaning

• Parameterized Projected Bellman Operator (AAAI 2024 · similarity 0.89)
• Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics (ICLR 2025 · similarity 0.88)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.88)
• Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.88)
• Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.88)
• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)