Research map / Vision-language and multimodal / Robot action and navigation
UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UP-VLA, a unified autoregressive model (built on Show-o) trained jointly with multimodal understanding, future image prediction, and action learning, evaluated on CALVIN and a real Franka robot.
Problem
VLA models built on VLMs capture high-level semantics but neglect low-level visual and spatial/dynamics information needed for precise robotic control.
Approach
Co-train a unified autoregressive model with three objectives (VQA-style understanding, future image token prediction, and action regression) so actions attend to both model-generated scene descriptions and predicted future visual tokens, using a flexible attention mask over task-specific prompt formats.
Key results
- On CALVIN ABC->D, UP-VLA reaches average length 4.08 versus 3.06 for GR-1 and 2.47 for Robo-Flamingo, the best reported result.
- On CALVIN ABCD->D, UP-VLA reaches average length 4.42 versus 4.21 for GR-1.
- Ablation: removing future prediction drops CALVIN ABC->D average length from 4.08 to 1.44.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual prediction to action learning (ablation) | UP-VLA-RT-2 · CALVIN ABC→D | Avg. Len: 1.44 -> 4.08 | Helped |
| Add multi-modal understanding training (ablation) | UP-VLA · CALVIN ABC→D | Avg. Len: 3.89 -> 4.08 | Helped |
| Add multi-modal understanding training (ablation) | UP-VLA · Real World Seen | Success rate: 0.85 -> 0.80 | Hurt |
| Add multi-modal understanding training (ablation) | UP-VLA · Real World Unseen | Success rate: 0.20 -> 0.58 | Helped |
| Add visual prediction pretraining on Bridge (ablation) | UP-VLA · CALVIN ABC→D | Avg. Len: 2.74 -> 4.08 | Helped |
| Add visual prediction pretraining on Bridge (ablation) | UP-VLA · Real World Seen | Success rate: 0.65 -> 0.80 | Helped |
| Add visual prediction pretraining on Bridge (ablation) | UP-VLA · Real World Unseen | Success rate: 0.30 -> 0.58 | Helped |
| Add visual prediction (ablation) | UP-VLA · CALVIN ABC→D | Avg. Len: 1.44 -> 4.08 | Helped |
| Add visual prediction (ablation) | UP-VLA · Real World Seen | Success rate: 0.65 -> 0.80 | Helped |
| Add visual prediction (ablation) | UP-VLA · Real World Unseen | Success rate: 0.35 -> 0.58 | Helped |
| Add MMU-generated descriptions to visual prediction prompts (ablation) | UP-VLA · CALVIN ABC→D | Avg. Len: 3.99 -> 4.08 | Helped |
| Add MMU-generated descriptions to visual prediction prompts (ablation) | UP-VLA · Real World Seen | Success rate: 0.80 -> 0.80 | No clear effect |
Surprising findings
- Removing future visual prediction causes the largest drop (4.08 to 1.44), while removing MMU lowers real-world seen-task performance only modestly, with unseen-object performance declining instead.
Limitations
- Object identification by the MMU component is sometimes inaccurate, attributed to data and backbone scale.
- Predicted frames can adopt background colors from the ABC training split rather than the D environment, indicating limited visual generation generalization.
- Real-world evaluation uses only 20 attempts per task and a small set of tasks.
Experimental setup
Datasets: Bridge LLaVA-tuning-665k CALVIN Franka real-world demonstrations (~2k)
Benchmarks: CALVIN ABC->D CALVIN ABCD->D real-world Franka manipulation
Metrics: average tasks completed in a row (Avg. Len) success rate over 20 attempts
Baselines: RT-1 Diffusion Policy Robo-Flamingo 3D-VLA UP-VLA-RT-2 Uni-Pi SuSIE GR-1 3D Diffuser Actor
Builds on: Show-o GR-1 RT-2 Phi-1.5 VQ-GAN
Closest papers by meaning
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.91)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.91)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.