Research map / Vision-language and multimodal / Robot action and navigation
DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamVLA, a VLA that predicts compact world knowledge (dynamic regions, depth, DINOv2/SAM semantics) via dream queries with block-wise structured attention and a DiT action head, reaching 4.44 avg. length on CALVIN ABC-D and 76.7% real-robot success.
Problem
Existing VLA models that forecast future images are inefficient because pixel forecasts are redundant and lack spatial (3D) and high-level semantic knowledge needed for action planning.
Approach
Replace full future-frame generation with predictions of dynamic regions (from CoTracker motion masks), monocular depth, and semantic features via separate learnable dream queries; mask attention among these sub-queries to avoid cross-type leakage, and decode actions with a diffusion transformer conditioned on an action query.
Key results
- Achieves 4.44 average length on CALVIN ABC-D, versus 4.29 for VPP and 4.28 for Seer.
- Reaches 76.7% overall real-world success on Franka tasks, versus 50.8% for Diffusion Policy and 35.0% for OpenVLA.
- Averages 92.6% on LIBERO across Spatial, Object, Goal, and Long suites.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic-region prediction to Vanilla VLA (ablation) | Vanilla VLA · CALVIN ABC-D | Avg. Len.: 3.64 -> 4.32 | Helped |
| Add depth prediction after dynamic-region prediction (ablation) | DreamVLA world-knowledge prediction · CALVIN ABC-D | Avg. Len.: 4.32 -> 4.40 | Helped |
| Add semantic prediction after dynamic-region and depth prediction (ablation) | DreamVLA world-knowledge prediction · CALVIN ABC-D | Avg. Len.: 4.40 -> 4.44 | Helped |
| Use future world-knowledge prediction instead of auxiliary reconstruction (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 4.14 -> 4.44 | Helped |
| Predict dynamic regions instead of optical flow (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 4.23 -> 4.44 | Helped |
| Replace vanilla causal attention with structured attention (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 3.75 -> 4.44 | Helped |
| Use separate modality queries instead of shared queries (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 4.17 -> 4.44 | Helped |
| Use 4 dream queries per modality instead of 9 (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 4.44 -> 4.32 | Hurt |
| Use 16 dream queries per modality instead of 9 (ablation) | DreamVLA · CALVIN ABC-D | Avg. Len.: 4.44 -> 4.33 | Hurt |
| Predict depth alone instead of dynamic-region knowledge (ablation) | DreamVLA knowledge-prediction variants · CALVIN ABC-D | performance | Hurt |
| Predict DINO or SAM features alone instead of dynamic-region knowledge (ablation) | DreamVLA knowledge-prediction variants · CALVIN ABC-D | performance | Hurt |
| Remove the DINO signal from the full knowledge-prediction model (ablation) | DreamVLA · CALVIN ABC-D | performance | No clear effect |
Surprising findings
- Predicting depth or DINO/SAM semantic features alone degrades performance below the vanilla VLA baseline, while dynamic-region prediction alone gives the largest gain.
- Predicting optical flow directly performs worse than predicting a binary dynamic-region mask derived from it.
Limitations
- Scope limited to parallel-gripper manipulation and RGB-centric data with limited geometric and material diversity.
- Depth predictions are coarse due to patch-level MAE-style decoding.
- Dynamic-region supervision depends on CoTracker motion heuristics; static regions are not directly supervised.
Experimental setup
Datasets: CALVIN DROID LIBERO LIBERO-90
Benchmarks: CALVIN ABC-D LIBERO real-world Franka pick/place/drawer tasks
Metrics: average task length success rate
Baselines: Roboflamingo Susie GR-1 3D Diffusor Actor OpenVLA RoboDual UniVLA Pi0 CLOVER UP-VLA RoboVLMs Seer
Builds on: CoTracker Depth-Anything DiT RoboVLMs Perceiver Resampler
Closest papers by meaning
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.92)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.91)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.91)
- CofactVLA: Deconfounding Vision-Language-Action Models via Counterfactual Intervention (NeurIPS 2026 · similarity 0.90)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.