Research map / Reinforcement learning / Offline RL
Planning, Fast and Slow: Online Reinforcement Learning with Action-Free Offline Data via Multiscale Planners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multiscale State-Centric Planners (MSCP), which pretrains goal-conditioned value and fast/slow action-free state planners from passive data, then uses them with pessimistic regularization to guide online RL.
Problem
Online RL with action-free (passive) offline data suffers from extrapolation error under distribution shift and difficulty handling long-horizon tasks with only single-scale planning.
Approach
Two state-centric planners at different horizons (fast 1-step, slow h-step) guide the low-level policy; the fast planner's intrinsic reward regularizes the policy to stay within dataset support, while the slow planner supplies subgoal-based advantages. Theory shows passive value learning under identifiable dynamics reduces effective horizon.
Key results
- On AntMaze-ultra-diverse, MSCP reaches 80.8 ± 10.3 normalized score versus 26.9 ± 17.4 for HIQL and 0.0 for pure online SAC.
- On CALVIN, MSCP achieves 65.0 ± 13.7 versus 44.5 ± 21.3 for HIQL in the passive setting.
- Pure online SAC fails on all AntMaze tasks (0.0 score) while passive pretraining methods succeed on several.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MSCP instead of HIQL | passive RL · AntMaze-medium-play | normalized score: 69.6 ± 9.7 -> 96.5 ± 3.2 | Helped |
| use MSCP instead of HIQL | passive RL · AntMaze-large-diverse | normalized score: 49.6 ± 30.4 -> 94.2 ± 3.0 | Helped |
| use MSCP instead of HIQL | passive RL · AntMaze-ultra-diverse | normalized score: 26.9 ± 17.4 -> 80.8 ± 10.3 | Helped |
| use MSCP instead of HIQL | passive RL · Kitchen-partial | normalized score: 13.1 ± 10.4 -> 33.0 ± 18.6 | Helped |
| use MSCP instead of HIQL | passive RL · CALVIN | normalized score: 44.5 ± 21.3 -> 65.0 ± 13.7 | Helped |
| use MSCP instead of HIQL | passive RL · Procgen-Maze-500 test | normalized score: 48.1 ± 8.7 -> 62.3 ± 9.3 | Helped |
| use MSCP instead of HIQL | passive RL · Procgen-Maze-1000 test | normalized score: 70.0 ± 7.4 -> 74.2 ± 7.8 | No clear effect |
| add fast SCP pessimistic guidance (ablation) | MSCP · AntMaze-large-diverse | normalized score: 49.6 ± 30.4 -> 94.2 ± 3.0 | Helped |
| add slow-SCP value guidance eAl (ablation) | MSCP · AntMaze-ultra-diverse | normalized score: 0.0 ± 0.0 -> 80.8 ± 10.3 | Helped |
| add slow SCP and condition the policy on its subgoal (ablation) | MSCP · AntMaze-large-diverse | normalized score: 0.0 ± 0.0 -> 94.2 ± 3.0 | Helped |
| fine-tune the SCPs with online and offline data (ablation) | MSCP · CALVIN | normalized score: 29.3 ± 28.5 -> 65.0 ± 13.7 | Helped |
| fine-tune the SCPs with online and offline data (ablation) | MSCP · Procgen-Maze-500 test | normalized score: 26.0 ± 0.04 -> 62.3 ± 9.3 | Helped |
Surprising findings
- Pure online SAC without pretraining fails completely on all AntMaze tasks, underscoring the value of passive pretraining.
- The fast planner's predicted next-state xy distance to actual next state increases during training, indicating it focuses on locomotion rather than navigation.
Limitations
- Fast SCP's next-state predictions are noisy for navigation, so the fast planner does not help long-horizon navigation directly.
- Performance is sensitive to exploration temperature.
- Passive-data theory relies on linear MDP and identifiable dynamics assumptions.
Experimental setup
Datasets: D4RL AntMaze D4RL Kitchen CALVIN Procgen Maze
Benchmarks: D4RL AntMaze D4RL Kitchen CALVIN Procgen Maze
Metrics: normalized score
Baselines: ICVF AF-Guide HIQL SAC HIQL offline oracle
Builds on: HIQL ICVF AF-Guide IQL
Closest papers by meaning
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.90)
- Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories (ICML 2024 · similarity 0.89)
- Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.