Research map / Reinforcement learning / MDPs and planning
An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policies
Mirco Mutti, Marcello Restelli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the Frobenius objective rather than the Infinity or Column Sum objectives in IDE³AL (ablation) | IDE³AL · Single Chain, Double Chain, and Knight Quest | state distribution entropy: Single Chain: 0.983 / 0.985 / 0.984; Double Chain: 0.967 / 0.961 / 0.970; Knight Quest: 0.720 / 0.837 / 0.850 | Mixed |
| use the Frobenius formulation rather than the alternative objective formulations during learning (ablation) | IDE³AL · Single Chain | state distribution entropy | Improved |
| iteratively alternate model estimation with optimization sweeps instead of collecting random-policy samples before offline optimization (ablation) | IDE³AL · Double Chain | transition model estimation error | Improved |
| use IDE³AL instead of CountBased | exploration policies · Double Chain | state entropy | Improved |
| use IDE³AL instead of MaxEnt | exploration policies · Double Chain and Single Chain | convergence speed | Improved |
| use CountBased instead of IDE³AL | exploration policies · Single Chain | exploration performance | No clear effect |
| use IDE³AL instead of MaxEnt | exploration policies · Knight Quest | state entropy | Improved |
| add a minimum action-probability constraint (ablation) | state-entropy exploration objective · Single Chain | state distribution entropy | Hurt |
| constrain the target transition matrix with ζ to prioritize faster mixing (ablation) | IDE³AL Infinity and Frobenius objectives · Single Chain | state distribution entropy | Mixed |
| use the uniform transition matrix as the target (ablation) | IDE³AL mixing objective · Single Chain | spectral gap | Improved |
| maximize entropy directly over the stationary state-action distribution rather than using a lower-bound formulation (ablation) | Dual formulation · Single Chain, Double Chain, and Knight Quest | state distribution entropy: Single Chain: 0.985 vs 0.984; Double Chain: 0.971 vs 0.970; Knight Quest: 0.859 vs 0.850 | No clear effect |
Closest papers by meaning
- Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimate (AAAI 2021 · similarity 0.89)
- Fast active learning for pure exploration in reinforcement learning (ICML 2021 · similarity 0.89)
- f-Policy Gradients: A General Framework for Goal-Conditioned RL using f-Divergences (NeurIPS 2023 · similarity 0.88)
- Fast Rates for Maximum Entropy Exploration (ICML 2023 · similarity 0.88)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.88)
- Scalable Online Exploration via Coverability (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
