Research map / Reinforcement learning / Policy gradient and actor-critic

State Entropy Regularization for Robust Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically and empirically that state entropy regularization yields robustness to spatially correlated reward and transition perturbations (beyond policy entropy), with limitations for kernel robustness, risk-averse CVaR, and sensitivity to rollout budget.

Problem

Whether entropy regularization over the state distribution provides provable robustness, and what kind, compared with policy entropy regularization, especially under spatially structured or correlated perturbations.

Approach

Derives via convex conjugate duality that state-distribution entropy regularization exactly solves a reward-robust RL problem with an explicit uncertainty set (LSE over states) and adversarial reward depending on global occupancy, unlike policy entropy whose adversary depends only on local policy; also gives a kernel-robustness lower bound and shows entropy regularization cannot fully solve kernel

Key results

• In MiniGrid, state-entropy-regularized agents significantly outperform policy-entropy and unregularized agents under spatially correlated wall perturbations.
• Under uniformly scattered small obstacles, both policy and state entropy improve over the unregularized agent, with policy entropy showing a slight advantage.
• With only 4 rollouts for entropy estimation, state entropy's robust performance can fall to policy entropy's level, while nominal performance stays largely unchanged.

Measured effects reported in the paper

Surprising findings

• Entropy regularization over the state distribution, when estimated on a single rollout, can yield deterministic behavior along a suboptimal longest path, degrading robustness.
• Entropy regularization (of any type) can arbitrarily degrade risk-averse CVaR performance.

Limitations

• Entropy regularization cannot fully solve kernel (transition) robustness in general, and the kernel lower bound is not claimed tight.
• Robustness benefits are sensitive to the number of rollouts used for state entropy estimation.
• Experiments are on small MiniGrid and MuJoCo tasks; state entropy does not consistently beat robust RL methods across all perturbation types.

Experimental setup

Datasets: MiniGrid MuJoCo Pusher MuJoCo Ant

Benchmarks: MiniGrid perturbations (walls, random obstacles, goal shift) MuJoCo Pusher with wall obstacle and reward shift MuJoCo Ant with vertical wall

Metrics: success rate nominal return robust return

Baselines: unregularized policy policy entropy regularization PRMDP NRMDP

Builds on: policy entropy regularization robust MDP state entropy exploration (Hazan et al.) k-NN entropy estimator

Closest papers by meaning

• Promoting Stochasticity for Expressive Policies via a Simple and Efficient Regularization Method (NeurIPS 2020 · similarity 0.91)
• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.90)
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.89)
• When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.88)