Research map / Reinforcement learning / Policy gradient and actor-critic
Bootstrapped Model Predictive Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BMPC, which imitates an MPC expert with a Gaussian network policy that also guides MPPI planning, pairs it with on-policy model-based TD value learning, and uses lazy reanalyze to cut re-planning cost.
Problem
Model-free policy learning in plan-based MBRL (TD-MPC2) struggles on high-dimensional locomotion tasks, yielding weak policies and inaccurate value estimates that degrade MPC planning.
Approach
Bootstrap MPC: the network policy imitates the MPC expert via a KL loss over Gaussian action distributions, guides MPPI planning in turn, and the value function is learned by on-policy model-based TD; a lazy reanalyze buffer reuses re-planned expert targets to reduce re-planning from 99% to 0.8% of samples.
Key results
- On 7 high-dimensional DMControl locomotion tasks, BMPC averages 90k steps to reach 90% of asymptotic reward versus 360k for TD-MPC2, a reported 300% data-efficiency gain.
- BMPC's network policy performs nearly on par with its MPC policy on DMControl, while TD-MPC2 shows a substantial gap on Dog and Humanoid tasks.
- Lazy reanalyze (k=10) reduces the reanalyze ratio from 99% to 0.8% and adds only 10-20% training wall-time; BMPC uses a 3M-parameter network versus TD-MPC2's 5M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BMPC instead of TD-MPC2 | continuous-control MPC · DMControl high-dimensional locomotion tasks (7-task average) | steps-to-solve: 360k steps -> 90k steps | Helped |
| Use BMPC instead of TD-MPC2 | continuous-control MPC · Dog Walk | time-to-solve: 2.03 -> 0.87 | Helped |
| Add expert-imitation policy to guide MPPI while retaining the original policy for value learning (ablation) | Variant 1 · Dog Run and Humanoid Run | performance | No clear effect |
| Also guide MPPI with the expert-imitation policy while retaining the original policy for value learning (ablation) | Variant 2 · Dog Run and Humanoid Run | performance | No clear effect |
| Replace max-Q policy learning with expert imitation and use that policy for value learning (ablation) | Variant 3 · Dog Run and Humanoid Run | performance | Helped |
| Use on-policy model-based TD-learning instead of Q-iteration (ablation) | BMPC · Dog Run and Humanoid Run | data efficiency | Helped |
| Increase lazy reanalyze frequency beyond interval k=10 (ablation) | BMPC · DMControl | performance | No clear effect |
| Increase lazy reanalyze frequency beyond interval k=10 (ablation) | BMPC · DMControl | computational overhead | Hurt |
| Increase policy standard deviation during lazy reanalyze (ablation) | BMPC · continuous-control tasks | overall performance | No clear effect |
| Use KL loss rather than log-probability loss for expert imitation (ablation) | BMPC policy learning · continuous-control tasks | policy standard-deviation imitation | Helped |
| Use log-probability loss rather than KL loss for expert imitation (ablation) | BMPC policy learning · continuous-control tasks | policy standard deviation | Hurt |
| Use lazy reanalyze with re-planning interval k=10 instead of re-planning every update | BMPC · continuous-control tasks | reanalyze ratio: 99% -> 0.8% | Helped |
Surprising findings
- Using the imitated policy to guide MPPI alone does not improve performance; the gain comes from using it to learn a better value function.
- Log-probability imitation loss led to large policy std and worse performance, so KL loss was preferred.
Limitations
- TD-horizon restricted to 1 because the world model cannot predict long horizons reliably.
- Evaluated only in online settings on DMControl and HumanoidBench; multi-task and offline settings not tested.
- Lazy reanalyze implementation runs single-threaded, so wall-time overhead is not minimized.
Experimental setup
Datasets: DMControl HumanoidBench
Benchmarks: DMControl HumanoidBench
Metrics: episode return steps-to-solve wall-time 95% confidence intervals
Baselines: SAC DreamerV3 TD-MPC2
Builds on: TD-MPC2 MPPI reanalyze (MuZero/EfficientZero) SAC DreamerV3
Closest papers by meaning
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.87)
- TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024 · similarity 0.86)
- Bisimulation Metric for Model Predictive Control (ICLR 2025 · similarity 0.85)
- World Models as Reference Trajectories for Rapid Motor Adaptation (NeurIPS 2025 · similarity 0.85)
- Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation (ICML 2024 · similarity 0.85)
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.