Research map / Reinforcement learning / Offline RL
Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reflect-then-Plan (RefPlan), a doubly Bayesian offline model-based planner that infers a VAE-based posterior over dynamics at test time and marginalizes over it when running sampling-based planning around an offline-learned prior policy.
Problem
Offline RL policies are fixed and conservative, and they degrade under high epistemic uncertainty from limited data, OOD initial states, and changed dynamics; existing MB planning methods handle this uncertainty poorly.
Approach
Recast offline MB planning as posterior inference over action sequences (control-as-inference with the offline policy as prior, estimated by importance sampling), and marginalize over a latent environment variable inferred from real-time history with a VariBAD-style encoder and a probabilistic ensemble decoder used for planning.
Key results
- Across 15 D4RL configurations for five prior policies, RefPlan outperforms LOOP in 10 (CQL), 7 (EDAC), 12 (MOPO), 9 (COMBO), and 12 (MAPLE) of 15 tasks, with average gain of 11.6% vs 5.3% for LOOP.
- Under OOD initialization (ME-trained, R-start), RefPlan boosts Walker2d performance by 16.4%, 31.4%, and 42.5% for COMBO, MAPLE, and CQL respectively.
- On HalfCheetah dynamics changes, RefPlan reaches 6238 on steep slope vs 2123 for the original policy and 3245 for LOOP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RefPlan planning instead of LOOP to the offline prior | CQL, EDAC, MOPO, COMBO, and MAPLE · D4RL MuJoCo tasks | normalized score: 11.6% average prior-policy gain for RefPlan; 5.3% for LOOP | Helped |
| Apply RefPlan planning rather than LOOP to CQL | CQL · Walker2d ME, OOD initialization | normalized score: 85.95 -> 92.61 | Helped |
| Apply RefPlan planning rather than LOOP to MAPLE | MAPLE · Hopper ME, OOD initialization | normalized score: 45.78 -> 42.50 | Hurt |
| Apply RefPlan planning rather than LOOP to MAPLE | MAPLE · HalfCheetah ME, OOD initialization | normalized score: 91.41 -> 91.84 | Helped |
| Use RefPlan instead of LOOP | MAPLE · HalfCheetah with changed dynamics | average return: 6088 -> 6190; 949.7 -> 1224; 2363 -> 2435; 3245 -> 6238; 2774 -> 3345 | Helped |
| Increase LOOP's sampling budget 16-fold | LOOP with CQL prior · Hopper MR | normalized score: LOOP (16x): 97.8 $\pm$ 1.1; RefPlan: 98.1 $\pm$ 0.5 | Helped |
| Train the prior using non-Markovian VAE dynamics (ablation) | MOPO · HalfCheetah M | normalized score: 42.8 -> 40.6 | Hurt |
| Train the prior using non-Markovian VAE dynamics and add RefPlan planning (ablation) | MOPO · HalfCheetah M | normalized score: 40.6 -> 66.45 | Helped |
| Train the prior using non-Markovian VAE dynamics (ablation) | MOPO · Hopper MR | normalized score: 90.3 -> 93.2 | Helped |
| Train the prior using non-Markovian VAE dynamics and add RefPlan planning (ablation) | MOPO · Hopper MR | normalized score: 93.2 -> 98.18 | Helped |
| Increase the number of latent samples in the outer expectation (ablation) | RefPlan · Hopper, HalfCheetah, and Walker2d on MR and FR | normalized score | Helped |
| Use non-Markovian VAE dynamics models for offline policy training rather than test-time planning (ablation) | MOPO and COMBO · Hopper, HalfCheetah, and Walker2d | normalized score | Mixed |
Surprising findings
- Giving LOOP a 16x larger sampling budget does not close the gap to RefPlan, suggesting gains are not just from extra compute.
- NM (Train) dynamics models used during MOPO training diverged or underperformed in several Hopper tasks.
Limitations
- Performance still drops notably under hill and gentle dynamics shifts in HalfCheetah.
- Evaluated only on single-task locomotion MuJoCo D4RL benchmarks.
- Uses exponential optimality likelihood and a sampling-based estimator rather than direct posterior sampling methods such as SIR, which may fail for multi-modal problems.
Experimental setup
Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2d
Benchmarks: D4RL HalfCheetah D4RL Hopper D4RL Walker2d
Metrics: normalized score average return median, IQM, mean, optimality gap (RLiable)
Baselines: LOOP CQL EDAC MOPO COMBO MAPLE MBOP
Builds on: VariBAD LOOP MBOP control-as-inference MPPI
Closest papers by meaning
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.91)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Bayesian Exploration Networks (ICML 2024 · similarity 0.88)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.