Research map / Reinforcement learning / Offline RL
Are Expressive Models Truly Necessary for Offline RL?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Recursive Skip-Step Planning (RSP), a lightweight supervised GCSL method using 2-layer MLPs that recursively predicts coarse skip-step sub-goals, matching or beating expressive-model offline RL methods on D4RL at far lower training and inference cost.
Problem
Goal-conditioned supervised learning for offline RL typically relies on large expressive models (Transformers, diffusion) for long-horizon sequential modeling, which incurs high computation and inference latency and suffers compounding step-wise errors.
Approach
Replace fine-grained step-by-step dynamics with recursively planned skip-step sub-goals (horizon K halved at each recursion level), each predicted by a shallow MLP conditioned on higher-level sub-goals, with a goal-conditioned MLP policy trained by supervised log-likelihood on relabeled data.
Key results
- RSP achieves 86.9 average on D4RL AntMaze (including Ultra), versus 67.6 for TT(+Q) and 61.2 for TAP(+G).
- RSP reaches 70.2 average on Kitchen, above HIQL-free baselines such as RvS(+G) at 55.9 and IQL at 48.7.
- RSP (N=1) trains in about 180 seconds with inference latency under 1 ms, versus much higher costs for expressive-model baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RSP with recursive skip-step planning instead of the compared baselines | offline RL · Antmaze navigation | normalized score | Helped |
| Use RSP instead of the compared baselines | offline RL · Kitchen-Partial | normalized score: 65.0 -> 76.5 | Helped |
| Use RSP instead of the compared baselines | offline RL · Kitchen-Mixed | normalized score: 67.7 -> 64.0 | Mixed |
| Use RSP instead of the compared baselines | offline RL · MuJoCo average | normalized score: 78.9 -> 76.4 | Mixed |
| Use RSP instead of the compared baselines | offline RL · Adroit average | normalized score: 67.0 -> 67.0 | No clear effect |
| Increase recursion depth from N=1 toward N=2 (ablation) | RSP · D4RL benchmark tasks | normalized score | Helped |
| Increase recursion depth beyond N=2 (ablation) | RSP · some D4RL tasks | normalized score | No clear effect |
| Reduce the skip-step horizon k (ablation) | RSP · Antmaze and Kitchen | normalized score | Hurt |
| Vary the skip-step horizon k (ablation) | RSP · Adroit and MuJoCo | normalized score | No clear effect |
| Increase skip-step horizon k (ablation) | RSP dynamics model · Antmaze-Ultra-Diverse rollout prediction | RMSE | Helped |
| Increase recursion depth N (ablation) | RSP dynamics model · Antmaze-Ultra-Diverse rollout prediction | RMSE | Helped |
| Use RSP with recursion depth N=1 | offline RL · Antmaze tasks | training time: 180 seconds | Helped |
Surprising findings
- Shallow 2-layer MLPs match or exceed Transformer and diffusion sequence models on D4RL, questioning the need for expressive models in offline RL.
- Horizon has minimal impact on short-horizon Adroit and MuJoCo tasks but reducing it degrades AntMaze and Kitchen performance.
Limitations
- Policy is a shallow 2-layer MLP, which limits how deep recursion and small horizons can be exploited; deeper recursion can accumulate errors and plateau beyond N=2.
- Choice of recursion depth and skip horizon requires tuning and is task-dependent (e.g., Kitchen needed dropout and smaller batch size due to limited data).
Experimental setup
Datasets: D4RL
Benchmarks: D4RL AntMaze D4RL Kitchen D4RL Adroit D4RL MuJoCo locomotion
Metrics: normalized D4RL score training time inference latency RMSE of sub-goal rollout predictions
Baselines: BC RvS CQL IQL HIQL DT TT TAP Diffuser
Builds on: goal-conditioned supervised learning Decision Transformer Trajectory Transformer TAP Diffuser
Closest papers by meaning
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.91)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.90)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.