Research map / Reinforcement learning / Policy gradient and actor-critic
Bootstrap Off-policy with World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOOM, which bootstraps an off-policy TD-MPC2-style world-model agent by aligning the policy to MPPI planner actions from the replay buffer via a Q-weighted, likelihood-free forward-KL loss.
Problem
Planning-augmented off-policy model-based RL suffers from actor divergence: data is collected by a planner (policy + MPPI) that differs from the policy, causing value distribution shift and unreliable policy improvement; planner action distributions are non-parametric with intractable likelihoods.
Approach
A bootstrap loop in which the policy initializes the MPPI planner and the policy is then trained to imitate the planner's buffered actions via a likelihood-free forward-KL alignment loss, weighted by softmax of Q-values so high-return planner actions dominate, added to the standard max-Q policy objective.
Key results
- On DMC Suite, BOOM averages 877.7 TAR vs 835.8 for BMPC (+5.0%) and 745.6 for TD-MPC2.
- On Humanoid-Bench, BOOM averages 820.6 TAR vs 555.6 for DreamerV3 (10M iters) and 511.7 for BMPC.
- Ablations on Dog-run show forward KL beats reverse KL and Q-weighting beats uniform weighting; alignment coefficient 0.1x-10x is stable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BOOM instead of BMPC | Planning-driven RL on DMC Suite · AVG. DMC Suite | TAR: 877.7 vs 835.8; +5.0% | Helped |
| use BOOM instead of DreamerV3 (10M) | RL on Humanoid Bench · AVG. H-Bench | TAR: 820.6 vs 555.6; +47.7% | Helped |
| add forward KL likelihood-free alignment instead of reverse KL alignment (ablation) | BOOM on Dog-run · Dog-run | TAR | Helped |
| add soft Q-weighting instead of uniform weighting (ablation) | BOOM on Dog-run · Dog-run | TAR | Helped |
| vary the alignment coefficient to 0.1× or 10× the default (ablation) | BOOM on Dog-run · Dog-run | TAR | No clear effect |
Surprising findings
- Reverse KL with a Gaussian surrogate of the planner distribution performs worse than the likelihood-free forward KL, suggesting surrogate likelihood approximations can be harmful.
Limitations
- Planning during data collection adds computational overhead, slowing training throughput.
- Relies on an accurate dynamics model; model errors over long rollouts can degrade planning and value estimates.
- Evaluated only on locomotion tasks in DMC and Humanoid-Bench with three seeds.
Experimental setup
Datasets: DeepMind Control Suite (dog and humanoid tasks) Humanoid-Bench (H1hand tasks)
Benchmarks: DeepMind Control Suite (7 dog/humanoid tasks) Humanoid-Bench (7 H1hand tasks)
Metrics: total average return training curves over 3 seeds
Baselines: SAC DreamerV3 TD-MPC2 BMPC LOOP TDM(PC)^2
Builds on: TD-MPC2 SAC AWAC MPPI
Closest papers by meaning
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.87)
- Policy-conditioned Environment Models are More Generalizable (ICML 2024 · similarity 0.85)
- Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.85)
- WPT: World-to-Policy Transfer via Online World Model Distillation (CVPR 2026 · similarity 0.84)
- Bootstrapped Model Predictive Control (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.