Research map / Reinforcement learning / Visual and representation RL
Off-policy Reinforcement Learning with Model-based Exploration Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoGE, a plug-in off-policy RL exploration module that generates utility-guided critical states with a classifier-guided diffusion model and expands them into transitions via a one-step latent world model, improving TD3 and DSAC on DMC and Gym locomotion tasks.
Problem
Active exploration (policy stochasticity) struggles in high-dimensional environments, while passive exploration (replay buffer prioritization or generative buffer augmentation) is limited by sample diversity and stays close to the original data distribution.
Approach
Generate under-explored critical states with a utility-guided (policy entropy or TD error) classifier-guided diffusion model, then synthesize their transitions with a one-step imagination world model, and mix these with replay samples in off-policy training without changing the base algorithm.
Key results
- On DMC, TD3+MoGE averages 161.9 vs TD3 93.4, and DSAC+MoGE reaches 817.7 vs DSAC 568.5 (+43.8%).
- On Gym, DSAC+MoGE averages 9135.5 vs DSAC 8301.0 (+10.0%); Humanoid-v3 reaches 12151.1 vs 10402.2.
- Ablation on Humanoid-run: latent dynamics (489) outperforms vanilla dynamics (408) for MoGE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoGE to DSAC | DSAC · AVG.DMC | TAR: 568.5±42.7 -> 817.7 ± 11.4; +43.8% | Helped |
| Add MoGE to DSAC | DSAC · AVG.Gym | TAR: 8301.0 ± 69.5 -> 9135.5 ± 122.1; +10.0% | Helped |
| Replace latent dynamics with vanilla dynamics (ablation) | MoGE · Humanoid-run | TAR: 489 ± 9 -> 408 ± 35 | Hurt |
| Replace latent dynamics with vanilla dynamics (ablation) | MoGE · Humanoid-walk | TAR: 892 ± 19 -> 684 ± 71 | Hurt |
| Replace latent dynamics with vanilla dynamics (ablation) | MoGE · Humanoid-stand | TAR: 907 ± 7 -> 785 ± 88 | Hurt |
| Use TD error rather than policy entropy for policy evaluation (ablation) | MoGE · Humanoid-run | TAR | Helped |
| Use policy entropy rather than TD error for policy improvement (ablation) | MoGE · Humanoid-run | TAR | Helped |
| Set guidance scale to 1 (ablation) | MoGE · Humanoid-run | TAR | Helped |
| Vary the generated-sample mix ratio from 0.1 to 0.5 (ablation) | MoGE · Humanoid-run | TAR: 0.1 to 0.5 | Mixed |
Surprising findings
- Walltime for DMC Humanoid-run is higher for MoGE (151657 s) than for DSAC (128153 s), i.e. a compute overhead the paper acknowledges.
- TD error utility is better for policy evaluation while policy entropy is better for policy improvement, suggesting utility choice should depend on update type.
Limitations
- Gains on some tasks rely on a small mixing ratio lambda; large lambda increases distribution mismatch, and importance sampling is only approximated.
- Model-based augmentation adds compute overhead relative to baselines (higher walltime).
- Theoretical stationarity assumption for the replay buffer distribution is strong and only empirically motivated.
Experimental setup
Datasets: DeepMind Control Suite OpenAI Gym
Benchmarks: DeepMind Control Suite (Humanoid-walk/stand/run, Quadruped-run/walk) OpenAI Gym (Walker2d-v3, Humanoid-v3, Ant-v3, Halfcheetah-v3, Swimmer-v3)
Metrics: Total Average Return (TAR) walltime
Baselines: TD3 DSAC TD3+PER DSAC+PER TD3+PGR DSAC+PGR SAC DDPG TRPO PPO TD-MPC2
Builds on: TD3 DSAC Prioritized Experience Replay Prioritized Generative Replay classifier-guided diffusion
Closest papers by meaning
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.87)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.86)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.