Research map / Reinforcement learning / Policy gradient and actor-critic

Decoupling regularization from the action space

ICLR 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that entropy-regularized RL over-regularizes states with more actions and proposes decoupled regularizers (range-normalized temperature) with static and dynamic temperature schemes, evaluated on DMC and a drug-design MDP.

Problem

Regularized RL (especially entropy-regularized) regularizes states inconsistently depending on the size of their action spaces, causing over-regularization and convergence issues, including in state-dependent action MDPs.

Approach

Define a regularizer's range over the action space and normalize the regularizer by it (decoupled regularizer), equivalently scaling the temperature per state by the log number of actions; also propose a dynamic automatic-temperature target based on a fraction of the range between uniform and minimum-entropy policies.

Key results

• On the drug design MDP, decoupled SQL achieves higher median reward than standard SQL and GFN throughout training, and finds more diverse high-quality modes.
• In the toy hypergrid MDP, standard SQL episode length grows very large for n>6 dimensions, while decoupled SQL stays close to the shortest path at a low temperature.
• On DMC with a static temperature of 0.25, decoupling does not worsen and often improves test reward compared to non-decoupled SQL across action scales.

Measured effects reported in the paper

Surprising findings

• Standard SQL over-regularizes states with more actions, causing the policy to prefer passing through these hub states in the drug design task.
• The SAC-style heuristic of target entropy equal to negative action dimension becomes infeasible for small action ranges, driving temperature to infinity and causing learning failure.

Limitations

• Theory of range-growth is restricted to standard regularizers over discrete action spaces; not all regularizers (e.g., positive-definite quadratic forms) are standard.
• Dynamic temperature alpha must be tuned; the recommended value 0.77 is tied to matching SAC results at actions in [-1,1].
• Scale invariance is not fully achieved since optimizers and losses are not scale invariant.

Experimental setup

Datasets: DeepMind Control suite hypergrid toy MDP fragment-based drug design MDP

Benchmarks: DeepMind Control suite fragment-based drug design MDP hypergrid toy MDP

Metrics: test reward episode length median reward number of modes

Baselines: soft actor-critic soft Q-learning GFlowNet trajectory balance path consistency learning

Builds on: soft actor-critic soft Q-learning path consistency learning trajectory balance GFlowNets

Closest papers by meaning

• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes (ICLR 2024 · similarity 0.88)
• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization (ICML 2024 · similarity 0.87)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.87)
• Demonstration-Regularized RL (ICLR 2024 · similarity 0.87)