Research map / Reinforcement learning / Policy gradient and actor-critic
When Maximum Entropy Misleads Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that maximum-entropy RL (SAC) can converge to suboptimal policies in precise control tasks because soft-Q landscapes mislead policy optimization, and proposes a diagnostic adaptive-entropy variant SAC-AdaEnt.
Problem
MaxEnt RL methods such as SAC often fail on performance-critical control tasks that require precise low-entropy policies, where non-MaxEnt methods like PPO succeed. The paper asks why, at convergence, entropy can bias the optimal policy.
Approach
Introduces an "entropy bifurcation extension" that adds intermediate states so that the MaxEnt-optimal policy at a target state can be made to follow an arbitrary distribution while the original MDP's optimal policy is preserved; supports this with a toy example, realistic control environments, and an adaptive-entropy SAC variant that switches to plain Q targets when soft and plain Q disagree.
Key results
- On a toy MDP, analytic and empirical results show SAC's mean converges to the wrong action region (A2) while PPO learns the correct one.
- Across five realistic control environments with five seeds, SAC underperforms PPO on Vehicle, Quadrotor and OpenCat, while SAC outperforms PPO on Acrobot, Obstacle2D and Hopper.
- The adaptive-entropy variant SAC-AdaEnt improves SAC on Vehicle and Quadrotor by switching to plain Q targets where soft and plain Q diverge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MaxEnt SAC rather than non-MaxEnt PPO | continuous-control policies · Vehicle, Quadrotor, and Opencat | reward | Hurt |
| use non-MaxEnt PPO rather than MaxEnt SAC | continuous-control policies · Vehicle, Quadrotor, and Opencat | reward | Helped |
| use MaxEnt SAC rather than non-MaxEnt PPO | continuous-control policies · Acrobot, Obstacle2D, and Hopper | reward | Helped |
| adaptively substitute plain Q targets when soft and plain Q landscapes differ | SAC · Vehicle | reward | Helped |
| adaptively substitute plain Q targets when soft and plain Q landscapes differ | SAC · Quadrotor | reward | Helped |
| auto-tune the entropy coefficient (ablation) | SAC · critical control environments | reward | Mixed |
| use entropy in the target Q values (ablation) | SAC · Hopper | reward | Helped |
| remove entropy encouragement | PPO · Obstacle2D | reward | Hurt |
| use MaxEnt exploration rather than PPO's incremental policy updates | policies · Acrobot | reward | Helped |
Surprising findings
- Auto-tuning SAC's entropy coefficient improves performance in some critical environments but still fails to surpass PPO.
- The authors state that the misleading effect of entropy can also explain SAC's advantages on benchmarks such as Hopper.
Limitations
- SAC-AdaEnt requires global estimation of Q values across the action space, which is hard to scale to high dimensions.
- Results are shown in simulated environments and a toy MDP only.
- The bifurcation construction is a theoretical existence argument rather than a practical attack or training method.
Experimental setup
Datasets: Vehicle simulator Quadrotor simulator OpenCat quadruped simulator Acrobot Obstacle2D Hopper (MuJoCo)
Benchmarks: Vehicle Quadrotor OpenCat Acrobot Obstacle2D Hopper
Metrics: episode return learning curves Q-value landscapes
Baselines: SAC PPO DDPG SAC with auto-tuned entropy
Builds on: Soft Actor-Critic PPO MaxEnt RL
Closest papers by meaning
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.90)
- Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow (NeurIPS 2024 · similarity 0.90)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.89)
- FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance (NeurIPS 2026 · similarity 0.89)
- S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.89)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.