Research map / Video / Video generation
Learning Visual Feature-Based World Models via Residual Latent Action
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Residual Latent Action (RLA), a compact latent learned from DINOv3 token residuals, and RLA-WM, which predicts RLA via flow matching for feature-space world modeling, then applies it to minimalist world action models and offline visual RL.
Problem
Visual feature-based world models using direct regression produce blurry or collapsed predictions in complex 3D manipulation, while generative modeling in high-dimensional DINO feature space is hard and expensive.
Approach
Learn a compact latent action from the residual between DINO tokens of two frames (single regression autoencoder), then run flow matching in this low-dimensional RLA space conditioned on current tokens and actions, decoding predicted RLA with the current frame to get future DINO tokens in one pass.
Key results
- RLA-WM achieves LPIPS 0.071, SSIM 0.931, DINO L1 0.030 on ManiSkill, beating DINO-WM (0.156/0.865/0.078) and Vid2World (0.199/0.705/0.084).
- RLA-WM uses 3.5T FLOPs versus 1.1P for Vid2World, roughly three orders of magnitude cheaper.
- RLA as latent action yields 35.6% average success over five ManiSkill tasks in actionless-video BC, versus 27.2% for BC-ResNet and 33.7% for AdaWorld; VLA-Adapter + RLA reaches 96.7% on LIBERO vs 96.1%.
Surprising findings
- RLA-WM hallucinates successes for out-of-distribution actions in PushT, predicting 84.8% success where the simulator gives 24.0% at noise scale 0.01.
- WMRL on Panda underperforms BC for Pull Cube and Pull Cube with Tool, with the authors attributing it to kinematics, occlusion, and limited action diversity.
Limitations
- Task-irrelevant background or large camera motion (humanoid, eye-in-hand) can pollute RLA.
- Single frame-pair RLA cannot handle occlusion or partial observability without history.
- Evaluation is on small-scale simulation and IWS data with only one real-robot task.
Experimental setup
Datasets: ManiSkill IWS LIBERO
Benchmarks: ManiSkill IWS LIBERO ManiSkill WMRL evaluation (750 episodes)
Metrics: LPIPS SSIM DINO L1 FLOPs success rate Spearman correlation
Baselines: DINO-WM RAE FM-WM Vid2World DINO CLS UniVLA AdaWorld LAPA VLA-Adapter BC-ResNet
Builds on: DINO-WM flow matching RAE PPO DINOv3
Closest papers by meaning
- Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.90)
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.89)
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.89)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.88)
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.88)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.