Research map / Reinforcement learning / Policy gradient and actor-critic

Decoupled Policy Actor-Critic: Bridging Pessimism and Risk Awareness in Reinforcement Learning

AAAI 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Actor-critic theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that pessimistic and optimistic actor-critic objectives (SAC, TD3, OAC) correspond to expected utility maximization under an exponential utility, and proposes DAC, an off-policy actor-critic with decoupled pessimistic and optimistic actors and automatic optimism/KL adjustment.

Problem

Pessimistic objectives used by actor-critic methods such as SAC and TD3 lack a theoretical basis, and it is unclear which class of policies they implement; the exploration-exploitation trade-off is also entangled in a single actor.

Approach

Interpret pessimistic and optimistic Q-bound objectives as certainty-equivalent value maximization under an exponential utility, then decouple exploration (optimistic actor) from TD learning and evaluation (pessimistic actor), with KL-based online adjustment of optimism and KL penalty.

Key results

• DAC outperforms benchmark model-free methods in sample efficiency and final performance across 30 locomotion and manipulation tasks (IQM with 10 seeds).
• In design ablations, BASE DAC scores 1.00 at RR=2 and 1.00 at RR=16, versus 0.08 (RR=2) for ONLY optimistic actor and 0.42 (RR=2) for NO KL REG.
• DAC requires about 15% additional wall-clock time over SAC and matches model-based TD-MPC on dog and humanoid domains with a smaller parameter count.

Measured effects reported in the paper

Surprising findings

• Using only the optimistic actor for both exploration and exploitation performs very poorly (0.08 at RR=2), showing the dual-actor design is essential.

Limitations

• Dual-actor framework incurs slightly higher memory and computation cost than SAC.
• Introduces three additional tunable hyperparameters (KL target, exploration multiplier, adjustment learning rates).
• Robustness of hyperparameters to more complex environments is uncertain.

Experimental setup

Datasets: DeepMind Control Suite MetaWorld MyoSuite

Benchmarks: DeepMind Control Suite MetaWorld MyoSuite

Metrics: IQM return success rate IQM with 95% CI

Baselines: TD3 SAC OAC TOP GPL TD-MPC DDPG

Builds on: Soft Actor-Critic TD3 Optimistic Actor-Critic TOP REDQ

Closest papers by meaning

• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.88)
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.88)
• Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.88)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
• Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.87)