Research map / Reinforcement learning / Visual and representation RL
How to Explore with Belief: State Entropy Maximization in POMDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes state entropy maximization to POMDPs by optimizing policy-gradient surrogates (MOE, belief-based MBE with belief regularization) over belief-averaged policies, with theory on proxy gaps and a "hallucination" effect.
Problem
Maximizing the entropy of true state visitation when the agent only receives partial, noisy observations (POMDP), where observation entropy can mismatch state entropy and optimal policies are non-Markovian and intractable.
Approach
Defines a memory-efficient belief-averaged policy class and a first-order policy gradient over belief-based proxy objectives (maximum believed entropy), plus a belief-entropy regularizer that mitigates the agent exploiting belief uncertainty ("hallucination").
Key results
- Belief-regularized MBE (Reg-MBE) outperforms non-regularized MBE and MOE across gridworld POMDP settings, with the largest gains under hard observation noise and stochastic transitions.
- MBE with belief-approximation noise up to variance 0.04 performs strikingly similar to exact-belief MBE, indicating robustness to approximate beliefs.
- MOE degrades markedly as observation quality worsens, while MBE remains robust to observation quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use belief-regularized MBE instead of unregularized MBE (ablation) | MBE policy gradient · POMDP gridworlds, varying observation quality | true state entropy | Helped |
| Use MOE rather than a belief-based proxy | POMDP entropy exploration · POMDP gridworlds with harder observation settings | true state entropy | Hurt |
| Use MBE rather than MOE | POMDP entropy exploration · POMDP gridworlds with diminishing observation quality | true state entropy | Helped |
| Optimize MSE directly when the POMDP specification is available | POMDP entropy exploration · Most tested POMDP gridworlds | true state entropy | Helped |
| Use belief-regularized MBE instead of MSE | POMDP entropy exploration · Limit cases with extreme observation–state disentanglement | true state entropy | Helped |
| Add belief-entropy regularization to MBE (ablation) | MBE policy gradient · POMDP gridworlds with stochastic transitions | true state entropy | Helped |
| Add belief-entropy regularization to MBE (ablation) | MBE policy gradient · POMDP gridworlds with hallucination | true state entropy | Helped |
| Use approximate beliefs with low approximation noise rather than exact beliefs (ablation) | MBE-based algorithms · POMDP gridworlds | true state entropy | No clear effect |
| Add belief-entropy regularization to MBE under belief perturbations (ablation) | MBE-based algorithms · POMDP gridworlds with belief approximation noise | true state entropy | Helped |
| Use belief-averaged policies instead of other considered policy classes (ablation) | POMDP entropy exploration · Considered POMDP instances | true state entropy | Helped |
| Use belief-averaged policies instead of Markovian policies over belief states (ablation) | POMDP entropy exploration · POMDP instances with manageable belief-state sets | true state entropy | Helped |
Surprising findings
- Plain MBE is severely hallucinated: the believed-state entropy overestimates true state entropy, an effect mitigated by belief regularization.
Limitations
- Experiments limited to small finite gridworld POMDPs; continuous domains left to future work.
- Assumes an oracle belief approximator with bounded L1 error; belief approximation not integrated into the pipeline.
- Proxy gaps depend on hallucination probability, making MBE a loose bound at high hallucination.
Experimental setup
Benchmarks: custom gridworld POMDPs
Metrics: true state entropy
Baselines: MSE (true-state oracle) MOE MBE Markovian observation policy Markovian belief policy
Builds on: REINFORCE maximum state entropy exploration (Hazan et al., 2019) maximum observation entropy
Closest papers by meaning
- Probing in the Dark: State Entropy Maximization for POMDPs (ICLR 2026 · similarity 0.93)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.89)
- Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.88)
- SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation (ICLR 2025 · similarity 0.87)
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.87)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.