Research map / Reinforcement learning / Visual and representation RL
DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DINO-WM, a latent world model that predicts future DINOv2 patch features from offline trajectories and plans zero-shot toward goal images with CEM-based MPC.
Problem
Learning world models from offline pre-collected trajectories that support test-time behavior optimization and task-agnostic reasoning without reward, expert demos, or inverse models.
Approach
Model dynamics entirely in the latent space of frozen pretrained DINOv2 patch features with a causal frame-level ViT predictor, decoupling the decoder from training, and plan to image goals via action-sequence optimization.
Key results
- On six environments DINO-WM reaches SR 0.98 Maze, 0.96 Wall, 0.92 Reach, 0.90 PushT, and CD 0.41 Rope and 0.26 Granular, beating IRIS, DreamerV3 and TD-MPC2 on the harder tasks.
- Using DINOv2 patch features instead of CLS or R3M/ResNet features raises PushT success from 0.44 (DINO CLS) to 0.90 and Wall from 0.58 to 0.96.
- On LPIPS for PushT, DINO-WM reaches 0.007 versus 0.039 for the best baseline (DINO CLS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DINO-WM instead of DreamerV3 | offline world-model planning · Reach | success rate: 0.64 -> 0.92 | Helped |
| use DINO-WM instead of IRIS | offline world-model planning · PushT | success rate: 0.32 -> 0.90 | Helped |
| use DINO-WM instead of IRIS | offline world-model planning · Rope | Chamfer Distance: 1.11 -> 0.41 | Helped |
| use DINO-WM instead of IRIS | offline world-model planning · Granular | Chamfer Distance: 0.37 -> 0.26 | Helped |
| increase training dataset size (ablation) | DINO-WM · PushT | success rate: 0.08 -> 0.92 | Helped |
| add causal attention mask (ablation) | DINO-WM with history h=2 · PushT | success rate: 0.36 -> 0.88 | Helped |
| add causal attention mask (ablation) | DINO-WM with history h=3 · PushT | success rate: 0.08 -> 0.92 | Helped |
| add causal attention mask (ablation) | DINO-WM with history h=1 · PushT | success rate: 0.76 -> 0.76 | No clear effect |
| omit decoder reconstruction loss from predictor training (ablation) | DINO-WM · PushT | success rate: 0.80 -> 0.92 | Helped |
| use DINO-WM instead of DreamerV3 | planning on unseen configurations · WallRandom | success rate: 0.76 -> 0.82 | Helped |
| use DINO-WM instead of DreamerV3 | planning on unseen configurations · PushObj | success rate: 0.18 -> 0.34 | Helped |
| use DINO-WM instead of IRIS | planning on unseen configurations · GranularRandom | Chamfer Distance: 0.86 -> 0.63 | Helped |
Surprising findings
- Adding a decoder reconstruction loss to the predictor lowered PushT success from 0.92 to 0.80.
- Without a causal attention mask, PushT success collapses as history length grows (0.36 at h=2, 0.08 at h=3).
Limitations
- Requires offline datasets with sufficient state-action coverage.
- Relies on ground-truth actions, which limits use of raw internet video.
- Plans only in action space, without hierarchical low-level control.
Experimental setup
Datasets: PointMaze random trajectories Wall random trajectories DeepMind Control Reacher trajectories Push-T replayed expert trajectories with noise Rope and Granular Nvidia Flex random-action trajectories
Benchmarks: PointMaze Wall Reacher Push-T Rope Granular WallRandom PushObj GranularRandom
Metrics: success rate Chamfer distance LPIPS SSIM
Baselines: IRIS DreamerV3 TD-MPC2 AVDC R3M ResNet DINO CLS
Builds on: DINOv2 IRIS DreamerV3 TD-MPC2 AVDC
Closest papers by meaning
- Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.89)
- Learning Visual Feature-Based World Models via Residual Latent Action (NeurIPS 2026 · similarity 0.89)
- SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators (NeurIPS 2026 · similarity 0.88)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.88)
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.88)
- Neural Motion Simulator Pushing the Limit of World Models in Reinforcement Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.