Research map / Reinforcement learning / Policy gradient and actor-critic
Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEow, a maximum-entropy RL framework that models the policy as a state-conditioned energy-based normalizing flow so the soft value has a closed form, merging policy evaluation and improvement into one loss.
Problem
Soft Q-learning and SAC-style MaxEnt RL for continuous actions need MCMC/variational sampling or Monte Carlo estimates of the soft value function, and actor-critic separation adds optimization error.
Approach
Use EBFlow so the policy's unnormalized density is the soft Q-function and its normalizing constant is the soft value in closed form; sampling via the inverse flow aligns the actor with the critic, so a single objective trains both. Adds learnable reward shifting and shifting-based clipped double Q-learning to handle numerical instability.
Key results
- MEow performs comparably to SAC and outperforms the other baselines (SQL, DDPG, TD3, PPO) in most of the five MuJoCo environments, with improvements over SAC on Ant-v4 and Humanoid-v4.
- On the Omniverse Isaac Gym tasks, MEow consistently outperforms SAC across Ingenuity, ANYmal, AllegroHand, and FrankaCabinet, and is superior on Ant and Humanoid.
- Ablation shows the vanilla formulation nearly fails to learn, while adding learnable reward shifting and then shifting-based clipped double Q-learning yields significant gains; the deterministic inference variant outperforms stochastic sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MEow instead of SAC | MuJoCo · Ant-v4 and Humanoid-v4 | total return | Helped |
| Use MEow instead of the compared baselines | MuJoCo · Five MuJoCo environments | total return | Mixed |
| Use MEow instead of SAC | Isaac Gym · Ant (Isaac) and Humanoid (Isaac) | total return | Helped |
| Use MEow instead of SAC | Isaac Gym · Ingenuity, ANYmal, AllegroHand, and FrankaCabinet | total return | Helped |
| Add learnable reward shifting (LRS) (ablation) | MEow (Vanilla) · Five MuJoCo environments | total return | Helped |
| Add shifting-based clipped double Q-learning (SCDQ) (ablation) | MEow (+LRS) · Five MuJoCo environments | final total return | Helped |
| Use a deterministic policy instead of a stochastic policy for inference (ablation) | MEow · Five MuJoCo environments | total return | Helped |
| Use additive coupling layers instead of affine coupling layers (ablation) | MEow · Five MuJoCo environments | total return | Helped |
| Add learnable reward shifting (LRS) (ablation) | SAC · Five MuJoCo environments | total return | No clear effect |
| Use different target smoothing parameter values across tasks (ablation) | MEow · Five MuJoCo environments | total return | Mixed |
Surprising findings
- Applying learnable reward shifting to SAC does not improve its performance.
- The vanilla MEow formulation shows negligible return growth, showing that the reward-shifting term is needed for numerical stability.
Limitations
- Deterministic inference relies on assumptions that the non-linear Jacobian determinants are input-independent, which may not hold for other flow architectures.
- Training is around 2.3x slower than SAC due to flow inference cost.
- Requires different target smoothing factors tau across tasks, so hyperparameter tuning is costly.
Experimental setup
Datasets: MuJoCo Omniverse Isaac Gym
Benchmarks: MuJoCo Hopper-v4 MuJoCo HalfCheetah-v4 MuJoCo Walker2d-v4 MuJoCo Ant-v4 MuJoCo Humanoid-v4 Omniverse Isaac Gym Ant Omniverse Isaac Gym Humanoid Ingenuity ANYmal AllegroHand FrankaCabinet
Metrics: total return training steps
Baselines: SQL SAC DDPG TD3 PPO Flow-SAC
Builds on: Soft Actor-Critic Soft Q-Learning EBFlow Clipped double Q-learning
Closest papers by meaning
- S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.91)
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.90)
- When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.90)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
- FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance (NeurIPS 2026 · similarity 0.88)
- Generative Actor-Critic with Soft Bridge Policies (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.