Research map / Reinforcement learning / Visual and representation RL
Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares model-free RL (GCIQL, HIQL, CRL, GCBC, HILP) and model-based planning (PLDM, a JEPA-style latent dynamics model with MPPI) for learning from reward-free offline navigation data of varying quality.
Problem
Learning to solve diverse goal-reaching and other tasks from large offline reward-free trajectory datasets of varying quality, layouts, and lengths; it is unclear how RL and optimal-control approaches compare in this setting.
Approach
Plan with a latent dynamics model trained with a reconstruction-free JEPA objective (similarity loss plus VICReg-style variance/covariance, time-similarity and inverse dynamics terms) using MPPI with an uncertainty-penalized goal-distance cost; the same model is reused for new tasks by changing only the cost.
Key results
- In Two-Rooms with 3M high-quality transitions, HIQL, GCIQL, HILP and PLDM nearly reach 100% success; PLDM is among the most sample-efficient methods, reaching about 80% with a few thousand transitions.
- On unseen PointMaze layouts trained on 5 maps, PLDM generalizes best while other methods fail; all baselines except PLDM degrade as layouts become more out-of-distribution.
- Replanning every 4 steps reduces PLDM's per-episode time from 16.0s to 4.8s with a normalized success rate of 0.95 (vs. 3.6s for GCIQL and 4.0s for HIQL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove variance loss from PLDM (ablation) | PLDM · Two-Rooms, optimal setting | success rate: 98.0 ± 1.5 -> 13.4 ± 9.2 | Hurt |
| Remove variance loss from PLDM (ablation) | PLDM · Diverse Maze, 5 training maps | success rate: 98.7 ± 2.8 -> 11.4 ± 6.5 | Hurt |
| Remove covariance loss from PLDM (ablation) | PLDM · Two-Rooms, optimal setting | success rate: 98.0 ± 1.5 -> 29.2 ± 4.4 | Hurt |
| Remove covariance loss from PLDM (ablation) | PLDM · Diverse Maze, 5 training maps | success rate: 98.7 ± 2.8 -> 7.8 ± 4.1 | Hurt |
| Remove temporal-similarity loss from PLDM (ablation) | PLDM · Two-Rooms, optimal setting | success rate: 98.0 ± 1.5 -> 71.0 ± 3.0 | Hurt |
| Remove temporal-similarity loss from PLDM (ablation) | PLDM · Diverse Maze, 5 training maps | success rate: 98.7 ± 2.8 -> 95.6 ± 3.2 | No clear effect |
| Remove inverse dynamics loss from PLDM (ablation) | PLDM · Two-Rooms, optimal setting | success rate: 98.0 ± 1.5 -> 98.0 ± 1.5 | No clear effect |
| Remove inverse dynamics loss from PLDM (ablation) | PLDM · Diverse Maze, 5 training maps | success rate: 98.7 ± 2.8 -> 75.5 ± 8.2 | Hurt |
| Replace VICReg objective with pixel reconstruction | PLDM latent-dynamics learning · Two-Rooms, good-quality data | success rate: 97.4 ± 1.3 -> 26.2 ± 13.9 | Hurt |
| Add inverse dynamics modeling | TD-MPC2 · Two-Rooms, good-quality data | success rate: 0.0 ± 0.0 -> 35.0 ± 0.0 | Helped |
| Increase replanning interval from every 1 step to every 4 steps (ablation) | PLDM · Two-Rooms, 200-step episode | time per episode: 16.0 ± 0.13 -> 4.8 ± 0.09 | Helped |
| Increase replanning interval from every 1 step to every 4 steps (ablation) | PLDM · Two-Rooms, 200-step episode | normalized success rate: 1.00 -> 0.95 | Hurt |
Surprising findings
- HILP and GCIQL generalize well to stitching and random-policy data, whereas most other model-free GCRL methods fail.
- TD-MPC2 collapses to 0% success in Two-Rooms; reconstruction-based latent dynamics (DreamerV3-style, pixel reconstruction) perform much worse than the reconstruction-free JEPA approach.
Limitations
- PLDM is about 4x slower than model-free baselines at per-step replanning, and performance drops with less frequent replanning.
- PLDM's performance drops on the stitching variant where doors are rarely traversed, and HILP is more data-hungry.
- Evaluated only on navigation tasks (point-mass, quadruped), not manipulation or partially observable settings.
Experimental setup
Datasets: Two-Rooms (self-generated) Diverse PointMaze (self-generated) Ant U-Maze exploratory data (self-generated)
Benchmarks: Two-Rooms Diverse PointMaze Ant U-Maze chasing (state-avoidance) task
Metrics: success rate average agent-chaser distance
Baselines: GCIQL HIQL HILP CRL GCBC DreamerV3 TD-MPC2
Builds on: JEPA VICReg OGBench MPPI HILP
Closest papers by meaning
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.90)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.90)
- Planning, Fast and Slow: Online Reinforcement Learning with Action-Free Offline Data via Multiscale Planners (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.