Research map / Vision-language and multimodal / Robot action and navigation
4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4D-VLA, which fuses depth-derived 3D coordinates and sampled historical RGB-D frames (via memory bank sampling) into a VLM-based VLA to reduce coordinate-system and state chaos during pretraining on DROID, and introduces the MV-Bench multi-view LIBERO benchmark.
Problem
Pretraining VLA models on diverse robot data is inefficient because single-frame RGB inputs under-specify the robot's coordinate frame and temporal state, producing dispersed conditional action distributions ("coordinate system chaos" and "state chaos").
Approach
Back-project depth into 3D world/robot coordinates and add a learnable positional embedding of these coordinates to visual features (spatial vision tokens), extend to multiple historical frames with temporal encodings, and select historical frames adaptively via a similarity-based memory bank sampler.
Key results
- On LIBERO, 4D-VLA reaches 88.6% average success rate versus 76.5% for OpenVLA (+12.1 points), with 79.1% on LIBERO-LONG.
- On MV-Bench In-View, 4D-VLA achieves 81.0% average success versus 52.2% for OpenVLA; Cross-View average 73.8% vs 50.5%.
- On real-world Franka tasks, the full model averages 85.63% versus 27.70% for OpenVLA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain with 4D-VLA | InternVL-4B · LIBERO average | success rate: 76.5 ± 0.6 -> 88.6 ± 0.3 | Helped |
| use 4D-VLA instead of OpenVLA | robot policies · MV-Bench In-View | success rate: 52.2 -> 81.0 | Helped |
| use 4D-VLA instead of OpenVLA | robot policies · MV-Bench Cross-View | success rate: 50.5 -> 73.8 | Helped |
| add coordinate encoding to pretrained Base VLA (ablation) | real-world robot policy · Task 3 precise placement | score: 40.00 -> 85.00 | Helped |
| add historical frames to pretrained Base VLA (ablation) | real-world robot policy · Task 2 robustness to distractors | score: 60.00 -> 77.50 | Helped |
| add coordinate encoding and historical frames to pretrained Base VLA (ablation) | real-world robot policy · Task 4 instruction following | score: 28.00 -> 80.00 | Helped |
| add proprioceptive tokens without pretraining or coordinate embeddings (ablation) | LIBERO-LONG policy · LIBERO-LONG | success rate: 29.5 -> 16.4 | Hurt |
| add 3D coordinate embeddings without pretraining (ablation) | LIBERO-LONG policy · LIBERO-LONG | success rate: 29.5 -> 36.6 | Helped |
| add pretraining and 3D coordinate embeddings (ablation) | LIBERO-LONG policy · LIBERO-LONG | success rate: 29.5 -> 45.0 | Helped |
| replace MLP action head with autoregressive action head (ablation) | LIBERO-LONG policy · LIBERO-LONG | success rate: 29.5 -> 28.1 | Hurt |
| replace MLP action head with diffusion action head (ablation) | LIBERO-LONG policy · LIBERO-LONG | success rate: 29.5 -> 27.0 | Hurt |
| use memory bank sampling instead of single-frame input (ablation) | InternVL2.5-4B · LIBERO-Spatial | success rate: 0.738 -> 0.866 | Helped |
Surprising findings
- Proprioceptive tokens hurt performance on LIBERO-LONG (16.4 vs 29.5 success without pretraining).
- The historical window size n matters much more than the number of sampled frames k, suggesting uniform sampling adds redundancy.
Limitations
- Requires RGB-D input and calibrated camera intrinsics/extrinsics, a hardware restriction.
- Depth from DROID is sparse or incomplete, handled by heuristic patch-level masking.
Experimental setup
Datasets: DROID LIBERO LIBERO-SPATIAL
Benchmarks: LIBERO MV-Bench RLBench (ARM4R protocol) real-world Franka tasks
Metrics: success rate FPS latency peak memory
Baselines: OpenVLA Octo DiffusionPolicy SpatialVLA TraceVLA UniAct SparseVLM FastV VLA-Cache ARM4R C2FARM-BC LLARVA
Builds on: OpenVLA InternVL HPT Octo
Closest papers by meaning
- ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.90)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- 4DP-QA: Scalable QA for 4D Perception in Vision Language Models (CVPR 2026 · similarity 0.89)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.88)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.88)
- RotVLA: Rotational Latent Action for Vision-Language-Action Model (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.