Research map / Reinforcement learning / Policy gradient and actor-critic
S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S²AC, a MaxEnt actor-critic that models the policy as a parameterized Stein Variational Gradient Descent sampler and derives a closed-form entropy estimate via change of variables.
Problem
Estimating the entropy of expressive energy-based policies in maximum-entropy RL is intractable; existing methods either use costly importance-sampling (SQL) or restrict policies to simple Gaussians (SAC).
Approach
Model the actor as a parameterized SVGD sampler (learned isotropic Gaussian initial distribution plus a few SVGD steps) whose update rule is invertible, so the change-of-variables formula gives a closed-form entropy using only first-order gradients and vector products.
Key results
- S²AC(φ,θ) outperforms SQL and SAC-NF across all five MuJoCo environments and SAC in four out of five.
- On the multi-goal environment, S²AC and SQL cover all modes for all entropy weights, while SAC covers them only at large alpha.
- Amortized S²AC has run-time similar to SAC and SQL (about 0.85 vs 0.72-0.84 on Hopper), while unamortized S²AC run-time grows with action dimension (3.267 on Hopper).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use parameterized Gaussian initialization for the SVGD sampler (ablation) | S²AC · MuJoCo benchmark | return | Helped |
| Amortize SVGD with a feed-forward policy for testing (ablation) | S²AC · MuJoCo benchmark | return | Hurt |
| Use parameterized Gaussian initialization and SVGD policy | S²AC · MuJoCo benchmark | return | Helped |
| Set SVGD kernel variance too small (ablation) | SVGD entropy estimate · Gaussian target entropy evaluation | entropy estimate correctness: σ=0.1 | Hurt |
| Set SVGD kernel variance too large (ablation) | SVGD sampler · Gaussian target entropy evaluation | convergence to target: σ=100 | Hurt |
| Use an intermediate SVGD kernel variance (ablation) | SVGD entropy estimate · Gaussian target entropy evaluation | entropy estimate correctness: σ∈{3, 5, 7} | Helped |
| Use SVGD rather than SGLD, DLD, or HMC | Entropy estimation · Gaussian target with ground truth entropy 3.41 | entropy estimate correctness: 3.41 | Helped |
| Parameterize the initial distribution of the SVGD sampler (ablation) | S²AC · Multi-goal environment | coverage of action modes | No clear effect |
| Increase the entropy weight α | S²AC · Multi-goal environment | frequency of reaching higher-entropy goals | Helped |
| Increase the entropy weight α | SAC · Multi-goal environment | frequency of reaching higher-entropy goals | No clear effect |
| Increase the entropy weight α | SQL · Multi-goal environment | frequency of reaching higher-entropy goals | No clear effect |
| Use a normalizing-flow policy instead of Gaussian policy | SAC-NF · Multi-goal environment | coverage of action modes | Mixed |
Surprising findings
- The change-of-variables entropy derived for LD and HMC samplers in prior work was counter-intuitively independent of sampler dynamics; the authors argue SVGD avoids this issue.
- Normalizing-flow SAC and SAC-based baselines fail to maximize expected future entropy in positive-reward multi-goal setups.
Limitations
- Invertibility of SVGD requires step size small relative to kernel variance, and the entropy formula relies on a small-step approximation.
- Unamortized S²AC run-time grows linearly with action dimension, and amortization trades some performance.
- Evaluation is limited to MuJoCo v2 environments and a custom 2D multi-goal task.
Experimental setup
Datasets: MuJoCo
Benchmarks: MuJoCo Hopper-v2 Walker2d-v2 HalfCheetah-v2 Ant-v2 Humanoid-v2 multi-goal environment
Metrics: episode return sample efficiency run-time rliable IQM / optimality gap
Baselines: SAC SQL SAC-NF DDPG PPO IAPO
Builds on: Soft Actor-Critic Soft Q-Learning Stein Variational Gradient Descent Normalizing flow SAC (SAC-NF) Amortized SVGD
Closest papers by meaning
- Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow (NeurIPS 2024 · similarity 0.91)
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.90)
- Generative Actor-Critic with Soft Bridge Policies (NeurIPS 2026 · similarity 0.89)
- When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.89)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
- Maximum Entropy Heterogeneous-Agent Reinforcement Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.