Research map / Reinforcement learning / Policy gradient and actor-critic

Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a temperature decoupling mechanism for vanishing-temperature entropy-regularized RL that provably converges to a reference-optimal policy and its return distribution, plus a convergent distributional algorithm for estimating optimal return distributions.

Problem

Entropy-regularized RL policies and return distributions do not converge to a well-characterized optimal policy as the temperature vanishes; existing distributional control iterates are unstable in the control setting.

Approach

Estimate action-values at a target temperature while acting with an amplified (decoupled) temperature alpha(tau) with alpha/tau -> 0; this converges to the optimality-filtered reference policy (uniform over optimal actions for uniform reference), and a soft distributional Bellman optimality operator yields convergent return distribution iterates.

Key results

• Theoretical: the temperature-decoupled policy pi^{tau,alpha} converges to the optimality-filtered reference policy pi^{*,ref} in TV (discrete actions) as tau->0 when alpha/tau->0.
• Theoretical: the soft distributional optimality iterates converge to return distributions of reference-optimal policies, with a contraction-like rate n*gamma^{n/p}.
• Numerically on a tabular MDP with alpha=tau^2, decoupled soft Q-learning policy converges to the uniform-over-optimal-actions policy while standard ERL collapses to delta_{a1} at x0.

Measured effects reported in the paper

Surprising findings

• Standard vanishing-temperature ERL can converge to a less diverse optimal policy than the reference-optimal one, even in tabular MDPs.
• Existing distributional control iterates (with the standard distributional optimality operator) do not converge in control settings, while the soft version does.

Limitations

• Policy convergence is qualitative; no rate or explicit choice of temperatures for epsilon-approximation of the return distribution.
• Return distribution algorithm (decoupled) stated for discrete action spaces; policy convergence for continuous actions only weak.
• Assumption requiring uniform lower bound on reference mass around optimal actions may be hard to guarantee in continuous action spaces.

Experimental setup

Benchmarks: tabular toy MDPs

Metrics: policy convergence as temperature vanishes return distribution estimates vs. reference-optimal baseline Wasserstein distance (theoretical)

Baselines: soft Q-learning (standard ERL at fixed temperature) distributional Bellman optimality operator

Builds on: soft Q-learning distributional Bellman operator categorical distributional RL maximum entropy RL

Closest papers by meaning

• Promoting Stochasticity for Expressive Policies via a Simple and Efficient Regularization Method (NeurIPS 2020 · similarity 0.90)
• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Decoupling regularization from the action space (ICLR 2024 · similarity 0.89)
• DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.89)