Research map / Reinforcement learning / Visual and representation RL
MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAMBA, a model-based meta-RL method built on DreamerV3 that encodes full meta-episodes with an RSSM and uses scheduled world-model horizons, improving sample efficiency and returns over VariBAD, HyperX, and Dreamer on meta-RL benchmarks.
Problem
Meta-RL methods are sample inefficient and mostly limited to low-dimensional task distributions, while model-based RL handles POMDPs well but has not been applied to meta-RL.
Approach
Use the Dreamer (DreamerV3) recurrent latent world model for meta-RL, modified to encode the full meta-episode history with the RSSM, sample latent indices uniformly for optimization, schedule the world-model horizon, and augment observations with reward and time step; theory shows decomposable task distributions favor local reconstruction.
Key results
- On Reacher-1, MAMBA reaches 655.5 ± 12.3 return versus 473.7 ± 27.5 for VariBAD and 552.0 ± 27.8 for Dreamer-Vanilla, with VariBAD and HyperX needing about 15x more frames to converge.
- On Rooms-4 MAMBA scores 136.7 ± 1.4 versus 88.1 ± 39.2 for VariBAD and 115.0 ± 6.7 for Dreamer-Vanilla; on Rooms-6 MAMBA scores 94.5 ± 1.2 versus 69.5 ± 6.3 for Dreamer-Vanilla.
- On Reacher-4, MAMBA reaches 77.7 ± 61.1 while VariBAD and HyperX obtain near zero returns; the visual Panda Reacher gives 137.6 ± 0.7 for MAMBA vs 111.1 ± 28.8 for Dreamer-Tune.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAMBA instead of Dreamer-Vanilla | DreamerV3 · Rooms-4 | total return: 119.8 $\pm$ 3.1 -> 136.7 $\pm$ 1.4 | Helped |
| Use MAMBA instead of Dreamer-Vanilla | DreamerV3 · Rooms-5 | total return: 96.5 $\pm$ 2.5 -> 113.1 $\pm$ 5.7 | Helped |
| Use MAMBA instead of Dreamer-Tune | DreamerV3 · Rooms-6 | total return: 71.0 $\pm$ 6.3 -> 94.5 $\pm$ 1.2 | Helped |
| Use MAMBA instead of Dreamer-Vanilla | DreamerV3 · Reacher-1 | total return: 552.0 $\pm$ 27.8 -> 655.5 $\pm$ 12.3 | Helped |
| Use MAMBA instead of Dreamer-Vanilla | DreamerV3 · Reacher-3 | total return: 183.6 $\pm$ 100.0 -> 325.0 $\pm$ 47.0 | Helped |
| Use MAMBA instead of Dreamer-Tune | DreamerV3 · EscapeRoom | total return: 73.2 $\pm$ 7.8 -> 73.9 $\pm$ 3.1 | No clear effect |
| Add local reconstruction instead of global reconstruction (ablation) | VariBAD · Rooms-4 | total return: 14.1 +- 3.0 -> 88.1+-39.2 | Helped |
| Add local reconstruction instead of global reconstruction (ablation) | VariBAD · Rooms-5 | total return: 0.9+-2.5 -> 15.1+-13.1 | Helped |
| Remove reward from the observation (ablation) | MAMBA · Rooms-3 | total return: 156.2$\pm$ 1.7 -> 11.5$\pm$ 1.4 | Hurt |
Surprising findings
- VariBAD with local (windowed) reconstruction outperforms vanilla global reconstruction on Rooms-4 and Rooms-5, supporting the decomposition theory.
- VariBAD and HyperX are highly sensitive to hyperparameters (e.g., one HyperX configuration gives unstable policies), while Dreamer-based methods work out of the box.
Limitations
- Runtime is higher than Dreamer variants because the RSSM unrolls over the entire meta-episode, requiring sequential encoding of many more transitions.
- Results are averaged over only four random seeds.
- Theoretical PAC bounds rely on KDE density estimation and mild smoothness assumptions and are not tightly connected to the deep RL implementation.
Experimental setup
Datasets: Point Robot Navigation Escape Room Rooms-N Reacher-N Panda Reacher Reacher with image observations Humanoid-Dir
Benchmarks: Point Robot Navigation Escape Room Rooms-3 Rooms-4 Rooms-5 Rooms-6 Reacher-1 Reacher-2 Reacher-3 Reacher-4 Panda Reacher
Metrics: meta-episode return environment steps to convergence wall-clock runtime
Baselines: VariBAD HyperX Dreamer-Vanilla Dreamer-Tune
Builds on: DreamerV3 VariBAD HyperX PPO
Closest papers by meaning
- DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.91)
- Meta-World+: An Improved, Standardized, RL Benchmark (NeurIPS 2025 · similarity 0.90)
- On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
- Meta-Q-Learning (ICLR 2020 · similarity 0.90)
- Counterfactual Bootstrap for Robust Meta-Reinforcement Learning (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.