Research map / Vision-language and multimodal / Robot action and navigation
Vision-Language-Action Instruction Tuning: From Understanding to Manipulation
Shuai Yang, Hao Li, Bin Wang, Yilun Chen, Yang Tian, Tai Wang, Hanqing Wang, Feng Zhao, Yiyi Liao, Jiangmiao Pang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add QA and captioning data to VLA-IT (ablation) | InstructVLA-Generalist · SimplerEnv-Instruct | average success rate: 41.7 -> 46.2 | Improved |
| enable explicit textual thinking before acting (ablation) | InstructVLA-Generalist · SimplerEnv-Instruct | success rate: 36.1% gain over direct execution | Improved |
| add language motion supervision (ablation) | InstructVLA · SimplerEnv | overall success rate: 9.3% improvement | Improved |
| add DINOv2 ViT encoder to the action expert (ablation) | InstructVLA action expert · SimplerEnv | performance: removal gives 50.0% drop | Improved |
| add FiLM to the ViT encoder (ablation) | InstructVLA action expert · SimplerEnv | performance: 15.3% further improvement | Improved |
| use MoE adaptation and two-stage training instead of full finetuning or autoregressive co-training (ablation) | InstructVLA · SimplerEnv | success rate: 12.5% improvement over Magma | Improved |
| remove MoE design while keeping training paradigm (ablation) | InstructVLA · SimplerEnv | manipulation and multimodal | Mixed |
| freeze action expert during VLA-IT instead of joint finetuning (ablation) | InstructVLA · SimplerEnv-Instruct | success rate | No clear effect |
| add VLA-IT data on both Bridge and Fractal vs Bridge only (ablation) | InstructVLA · SimplerEnv-Instruct | overall success rate: 15.6 -> 21.7 -> 46.0 | Improved |
| finetune OpenVLA with VLA-IT data on top of multimodal data (ablation) | OpenVLA · SimplerEnv-Instruct situated reasoning | success rate: 19.5 -> 17.4 | No clear effect |
| add GPT-4o as system-2 instruction rewriter to OpenVLA (ablation) | OpenVLA + VL · SimplerEnv-Instruct | average success rate: 23.9 -> 35.6 | Improved |
| increase number of latent action tokens (16 to 128) (ablation) | InstructVLA · SimplerEnv | robot performance | Mixed |
Closest papers by meaning
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.93)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.93)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.93)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.92)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
