Research map / Reinforcement learning / Policy gradient and actor-critic

Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Actor-critic theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Langevin Soft Actor-Critic (LSAC), a model-free continuous-control RL method that learns a distributional critic via aSGLD (approximate Thompson sampling) with parallel critics and diffusion-synthesized transitions refined by Q action gradients.

Problem

Actor-critic methods for continuous control have poor sample efficiency because they lack principled exploration; naive Langevin-based Thompson sampling for Q-functions does not scale to multidimensional continuous action spaces and suffers from slow mixing and low update-to-data ratios.

Approach

Combine distributional Max-Ent critic learning with adaptive SGLD posterior sampling of critic parameters, run multiple same-temperature LMC chains (simplified parallel tempering) to cover multimodal Q posteriors, and augment critic batches with diffusion-generated transitions whose actions are refined by Q action gradients.

Key results

• On MuJoCo, LSAC achieves the highest maximum average return on 5 of 6 tasks (e.g., HalfCheetah 17948 ± 1724 vs DSAC-T 12703 ± 1711); DSAC-T is slightly better on Humanoid (9028 vs 8545).
• On 12 DMC hard-exploration tasks, LSAC outperforms the listed baselines in 9 of 12 tasks (e.g., Reacher Hard 981 vs DSAC-T 11).
• In PointMaze and AntMaze, LSAC discovers multiple paths to both goals, while most baselines find only one path or fail (except DIPO, which also finds multiple paths).

Measured effects reported in the paper

Surprising findings

• Replacing the distributional critic with a standard SAC critic causes a significant performance drop, and the distributional critic shows a smaller, near-constant underestimation bias.
• Removing the diffusion Q action gradient causes only a marginal decline on HalfCheetah but a larger drop on Ant and Swimmer.

Limitations

• Parallel tempering is simplified: all replicas share the same temperature and no replica exchange is performed.
• Performance is sensitive to the LMC step size and the adaptive bias factor a.
• Humanoid results are slightly below DSAC-T; evaluation limited to state-based MuJoCo and selected DMC tasks.

Experimental setup

Datasets: MuJoCo DeepMind Control Suite D4RL

Benchmarks: MuJoCo HalfCheetah-v3 MuJoCo Ant-v3 MuJoCo Swimmer-v3 MuJoCo Walker2d-v3 MuJoCo Hopper-v3 MuJoCo Humanoid-v3 DeepMind Control Suite (12 hard tasks) PointMaze_Medium-v3 AntMaze-v4

Metrics: maximum average return state coverage normalized Q estimation bias exploration density

Baselines: DSAC-T DIPO SAC TD3 PPO TRPO REDQ QSM DrQ-v2 Dreamer

Builds on: SAC DSAC-T LMCDQN SynthER DIPO

Closest papers by meaning

• Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo (ICLR 2024 · similarity 0.90)
• Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning (ICML 2024 · similarity 0.89)
• Langevin Policy for Safe Reinforcement Learning (ICML 2024 · similarity 0.89)
• Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.89)
• Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.88)
• ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.88)