Research map / Vision-language and multimodal / Robot action and navigation
3D-VLA: A 3D Vision-Language-Action Generative World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-VLA, a 3D vision-language-action generative world model built on a 3D-LLM-style BLIP2-FlanT5 backbone with interaction tokens and aligned diffusion decoders that generate goal images, depths and point clouds, trained on a 2M-sample 3D embodied instruction dataset.
Problem
Existing vision-language-action models rely on 2D inputs and learn direct perception-to-action mappings without modeling world dynamics or future states, and embodied datasets lack 3D annotations.
Approach
Combine a 3D-aware LLM with special interaction tokens (object, location, scene, action) and embodied diffusion models for RGB-D and point cloud goal generation, bridged to the LLM by a transformer projector, so the model imagines goal states before predicting actions.
Key results
- On held-in embodied QA, 3D-VLA reaches BLEU-4 26.80 and EM@1 24.53, versus 15.48 and 15.35 for the BLIP2 FlanT5-XL baseline trained on the same data.
- On CALVIN long-horizon chaining, 3D-VLA completes 44.7% of single tasks and 8.1% of three in a row, versus 28.2% and 0.3% for MCIL.
- Localization on held-in robotics data reaches IoU 29.33 and Acc@25 42.26, above CoVLM (19.81 / 25.39) and Kosmos-2 (10.92 / 12.73), all with GT depth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D-VLA instead of BLIP2 FlanT5 XL | Embodied QA · Embodied QA | BLEU-1: 37.31 -> 48.34 | Helped |
| Use 3D-VLA instead of BLIP2 FlanT5 XL | Task Caption · Task Caption | BLEU-1: 22.05 -> 55.69 | Helped |
| Use 3D-VLA instead of BLIP2 FlanT5 XL | What-if QA · What-if QA | BLEU-1: 28.23 -> 53.09 | Helped |
| Use 3D-VLA instead of BLIP2 FlanT5 XL | Dense Caption · Dense Caption | BLEU-1: 36.17 -> 51.90 | Helped |
| Use 3D-VLA instead of CoVLM with ground-truth depth | Held-in robotics localization · Held-in robotics datasets | IoU: 19.81 -> 29.33 | Helped |
| Use 3D-VLA instead of Instruct-P2P* | RGB goal image generation · Open-X test sets | PSNR: 16.67 -> 17.21 | Helped |
| Use 3D-VLA instead of Point-E* | Point-cloud goal generation · Open-X test sets | P-FID: 5.241 -> 4.796 | Helped |
| Use 3D-VLA instead of LanCon-Learn w/ His. | RLBench action planning · RLBench: Put Knife | success rate: 32.2 -> 68 | Helped |
| Use 3D-VLA instead of LanCon-Learn w/ His. | RLBench action planning · RLBench: Take Umbrella | success rate: 50.8 -> 52 | Helped |
| Use 3D-VLA instead of LanCon-Learn w/ His. | RLBench action planning · RLBench: Pick up Cup | success rate: 44.2 -> 40 | Hurt |
| Use 3D-VLA instead of MCIL | CALVIN action planning · CALVIN: tasks completed in a row (1) | tasks completed: 28.2 -> 44.7 | Helped |
| Add predicted bounding boxes to the input prompt (ablation) | 3D-VLA RGB goal image generation · Open-X test sets | PSNR: 17.02 -> 17.21 | Helped |
Surprising findings
- 3D-LLM performs poorly on robotic reasoning tasks in zero-shot transfer (e.g., EM@1 0.00 on embodied QA), which the authors take as evidence that robotics-specific 3D data is needed.
- Pretrained 3D-LLM weights were not loaded because its training data (objects and indoor scenes) did not match the embodied setup; the model was built on BLIP2-FlanT5 instead.
Limitations
- Evaluation is on held-in datasets only, with zero-shot and held-in comparisons against baselines trained on the same data.
- Action planning uses open-loop control and is compared on small task sets in RLBench and CALVIN.
- Dataset construction relies on automatic tools (ZoeDepth, Grounded-SAM, ChatGPT) whose errors are not quantified.
Experimental setup
Datasets: Open-X Embodiment subsets Dobb-E RH20T RLBench CALVIN Epic-Kitchens HOI4D RoboVQA RT-1 Jaco Play
Benchmarks: RoboVQA RLBench CALVIN Open-X held-in tasks
Metrics: BLEU-1..4 METEOR ROUGE-L EM@1 IoU Acc@25 Acc@50 PSNR CLIP similarity SSIM
Baselines: 3D-LLM BLIP2 OpenFlamingo LLaVA Kosmos-2 CoVLM Instruct-P2P SuSIE NeXT-GPT Point-E LanCon-Learn MCIL
Builds on: 3D-LLM BLIP-2 Stable Diffusion Point-E DreamLLM
Closest papers by meaning
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.91)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.91)
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.91)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.90)
- Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.90)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.