Research map / Reinforcement learning / Policy gradient and actor-critic
SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEMDICE, an off-policy state entropy maximization method that optimizes over stationary state-action distributions via a DICE-style convex minimization, yielding a single stationary SEM policy from arbitrary replay data.
Problem
Unsupervised RL pre-training via state entropy maximization (SEM) is typically on-policy (sample-inefficient) or off-policy but biased, since existing methods estimate the entropy of the replay buffer rather than the target policy's stationary state distribution.
Approach
Formulate SEM as a regularized concave program over state-action stationary distributions with an f-divergence trust-region to the dataset distribution and a marginalization constraint; the Lagrangian reduces to a single convex minimization over (nu, mu) solvable from off-policy (s,a,s') samples, after which the policy is extracted by i-projection.
Key results
- In tabular MDPs, SEMDICE is the only method that converges to an optimal SEM policy (normalized entropy approaching 1) while baselines do not directly maximize target-policy state entropy.
- On URLB (Walker, Quadruped, Jaco Arm), SEMDICE achieves the highest state entropy across domains and reaches maximum state entropy with about 100K steps in Walker.
- In URLB fine-tuning, SEMDICE significantly outperforms all data-based and knowledge-based baselines, though it underperforms the competence-based CIC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEMDICE instead of the policy-gradient baselines | state-entropy maximization on randomly generated tabular MDPs · Randomly generated tabular MDPs | normalized policy entropy | Helped |
| Use CB-S rather than CB-SA | policy-gradient count-based baselines · Randomly generated tabular MDPs | normalized policy entropy | Helped |
| Use PB-S to maximize replay-buffer state entropy | off-policy SEM baseline · Randomly generated tabular MDPs | replay-buffer state entropy | Helped |
| Use on-policy policy gradients instead of SEMDICE | SEM policy learning · Randomly generated tabular MDPs | samples required to reach near-optimal entropy: × 100 more samples | Hurt |
| Use SEMDICE rather than existing SEM methods | state visitation policy · MountainCar (Continuous) | state entropy | Helped |
| Use SEMDICE rather than the compared URLB pretraining methods | reward-free pretraining · URLB Walker, Quadruped, and Jaco Arm | state entropy | Helped |
| Use SEMDICE rather than data-based and knowledge-based baselines | downstream-task fine-tuning · URLB downstream tasks | fine-tuning performance | Helped |
| Use SEMDICE rather than CIC | downstream-task fine-tuning · URLB downstream tasks | fine-tuning performance | Hurt |
| Vary alpha within a reasonable range (ablation) | SEMDICE · Tabular MDPs and ContinuousMountainCar | state entropy | No clear effect |
| Use very small alpha (ablation) | SEMDICE · Tabular MDPs and ContinuousMountainCar | numerical stability | Hurt |
| Vary the f-divergence (ablation) | SEMDICE · Tabular MDPs and ContinuousMountainCar | state entropy | No clear effect |
Surprising findings
- SEMDICE still converges to an optimal SEM policy when the dataset is collected by a uniform random policy.
- SEMDICE underperforms CIC, a competence-based URL method, in URLB fine-tuning.
Limitations
- Relies on kNN particle-based estimate of -log d^D(s), which introduces approximation bias.
- Convergence guarantee for the practical policy-based algorithm is left as future work.
- Evaluated only on state-based (non-pixel) domains; representation learning not included.
Experimental setup
Datasets: URLB Walker URLB Quadruped URLB Jaco Arm randomly generated tabular MDPs MountainCar (Continuous) from Gymnasium
Benchmarks: URLB MountainCar (Continuous) state coverage randomly generated 20-state 4-action tabular MDPs
Metrics: normalized state entropy IQM and aggregate statistics via rliable fine-tuning returns
Baselines: RND ProtoRL APT MEPOL DIAYN CIC Disagreement MaxEnt CB-SA CB-S PB-S
Builds on: DICE URLB ProtoRL MaxEnt particle-based entropy estimation
Closest papers by meaning
- Towards Principled Unsupervised Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.88)
- Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
- How to Explore with Belief: State Entropy Maximization in POMDPs (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.