Research map / Reinforcement learning / Visual and representation RL
Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Dyna-style model-based RL (MBPO, ALM) matches or beats SAC on OpenAI Gym but underperforms or fails to improve beyond random policy on DeepMind Control, and probes model error, perfect models, plasticity resets and critic divergence while releasing a faster JAX MBPO implementation.
Problem
Dyna-style off-policy model-based RL (MBPO, ALM) shows strong gains on OpenAI Gym proprioceptive tasks but often fails to learn in DeepMind Control Suite tasks, even though the two suites use similar tasks and the same MuJoCo physics; it is unclear whether this gap is fundamental.
Approach
Systematically compares Dyna-style enhancements against their no-Dyna base (SAC for MBPO, TD-loss DDPG for ALM) across Gym and DMC, then tests remedies (hyperparameter sweeps, perfect model oracle, layer norm, model resets, actor/critic plasticity resets) and provides a much faster JAX MBPO implementation.
Key results
- On six DMC tasks, MBPO fails to make policy improvement over random initialization across 6 seeds, while SAC learns; the same gap appears for ALM vs. ALM without Dyna across six DMC tasks.
- Model error on DMC hopper tasks converges above 100% and other failing DMC tasks above 25%, versus below 25% in all six Gym tasks.
- Even with a perfect 1-step model, MBPO does not consistently outperform SAC on DMC; the JAX implementation reduces compute to about 4 GPU-days versus roughly 104 days estimated for the original, a reported 26x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 1-step synthetic rollouts to SAC | MBPO · OpenAI Gym environments | episodic return | Helped |
| Add Dyna-style synthetic rollouts to SAC | MBPO · DeepMind Control Suite environments | episodic return | Hurt |
| Use Dyna-style synthetic rollouts | ALM · DeepMind Control Suite environments | episodic return | Hurt |
| Increase the synthetic-to-real data ratio (ablation) | MBPO · DMC hopper-stand and humanoid-stand | episodic return | Hurt |
| Use a perfect rather than learned predictive model (ablation) | MBPO · DMC environments | episodic return | Mixed |
| Tune predictive-model hyperparameters (ablation) | MBPO · DMC hopper-stand and humanoid-stand | episodic return | No clear effect |
| Apply Layer Normalization to the critic (ablation) | MBPO · DMC environments | episodic return | No clear effect |
| Periodically reset predictive-model parameters (ablation) | MBPO · DMC environments | model accuracy | No clear effect |
| Periodically reset actor, critic, target critic, and temperature (ablation) | MBPO · DMC quadruped and humanoid-walk tasks | episodic return | Helped |
| Periodically reset actor, critic, target critic, and temperature (ablation) | MBPO · Other DMC tasks | episodic return | No clear effect |
| Periodically reset actor, critic, target critic, and temperature (ablation) | SAC versus MBPO · DMC environments | episodic return | Hurt |
Surprising findings
- Even with access to a perfect dynamics model, MBPO does not consistently outperform its base SAC on DMC.
- SAC with periodic resets significantly outperforms MBPO with the same resets on most tasks, and hyperparameter sweeps over 174 configurations gave no consistent MBPO improvement on hopper-stand and humanoid-stand.
Limitations
- Does not identify a definitive cause of the DMC gap; the authors acknowledge the study cannot exhaustively test all possible factors.
- Experiments concentrate on MBPO and ALM with fixed hyperparameters; DreamerV3 is discussed but not tested in the same setup.
- Hyperparameter sweeps were limited to subsets of parameters on two tasks with single or three seeds.
Experimental setup
Datasets: OpenAI Gym MuJoCo tasks DeepMind Control Suite tasks
Benchmarks: OpenAI Gym MuJoCo DeepMind Control Suite
Metrics: episodic return sample efficiency wall-clock time percent model error mean Q-value
Baselines: SAC ALM ALM without Dyna original PyTorch MBPO DreamerV3
Builds on: MBPO SAC ALM DDPG DreamerV3
Closest papers by meaning
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Scaling Offline RL via Efficient and Expressive Shortcut Models (NeurIPS 2025 · similarity 0.86)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.86)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.85)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.