Research map / Reinforcement learning / Policy gradient and actor-critic

Generative Actor-Critic with Soft Bridge Policies

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SoftGAC, a one-pass stochastic bridge actor in pre-tanh latent space whose path-wise KL to a high-entropy reference reduces to sampled transition control energy, enabling MaxEnt actor-critic with low-latency generative policies.

Problem

Maximum-entropy online RL with expressive generative (diffusion/flow) policies is hard because marginal action densities are unavailable (so entropy must be bounded or approximated), and iterative shared-parameter samplers make inference costly and destabilize BPTT-based actor updates.

Approach

Model the actor as a short chain of K lightweight step-specific Gaussian transitions from a fixed base latent to a terminal pre-tanh latent, and lift the MaxEnt objective to a path-space relative entropy against a finite-step high-entropy reference bridge; this KL decomposes exactly into sampled local Gaussian control energies used directly as the actor regularizer.

Key results

• SoftGAC attains higher or competitive IQM returns than diffusion and flow-matching baselines on 8 hard continuous-control tasks, with largest gains on Humanoid Run, Dog Run, H1 Hurdle and H1 Stair.
• SoftGAC uses about 61-75 microseconds per action on CPU (Apple M3 Pro), in the low-latency range of one-step flow baselines and far below high-NFE diffusion baselines.
• Setting alpha=0 (removing the path-space soft regularizer) consistently hurts Humanoid Run, Humanoid Walk, Dog Run and H1 Hurdle.

Limitations

• Endpoint connection to the uniform action prior depends on finite-step reference discretization and base law (endpoint bias O(d_a/K^2)).
• Introduces architecture choices such as bridge depth, base distribution and target control-energy budget.
• Global optimality result applies to the unrestricted path-law family, not the neural Gaussian Markov actor trained by SGD.

Experimental setup

Datasets: DeepMind Control Suite HumanoidBench

Benchmarks: DeepMind Control Suite HumanoidBench

Metrics: interquartile mean return per-action inference latency actor parameter count

Baselines: FLAC DIME FlowRL QSM QVPO CrossQ-SAC

Builds on: soft actor-critic CrossQ FLAC DIME

Closest papers by meaning

• Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.92)
• S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.89)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.88)
• DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
• Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow (NeurIPS 2024 · similarity 0.87)