Research map / Vision-language and multimodal / Robot action and navigation
VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLAS, an end-to-end vision-language-action model that takes raw speech instructions (no separate ASR) built on LLaVA, with a voice retrieval-augmented generation module for personalized manipulation, plus new SQA and CSI speech datasets.
Problem
Current VLAs accept only text instructions; cascaded ASR pipelines add complexity, propagate errors, and lose non-semantic speech cues (e.g., voiceprint) needed for customized robot tasks.
Approach
Integrate a Whisper speech encoder with a LLaVA-based VLA so raw speech is tokenized alongside images, trained in three stages (speech alignment, speech QA, robot fine-tuning), and augment inputs with a voiceprint-keyed retrieval module (Voice RAG) that injects personal knowledge.
Key results
- On CALVIN ABCD/D with speech instructions, VLAS reaches Len 3.70 versus 3.13 for VLA with Whisper ASR and 3.41 for RoboFlamingo with ASR.
- On the customized benchmark, VLAS with speech averages 86.5% success versus 19.2% for the text-only VLA; removing Voice RAG drops VLAS to 16.0%.
- VLAS-Base scores 72.2 on SQA-I and 51.1 on VizWiz, and matches LLaVA v1.5 on GQA (62.0) and nearly on VQA-v2 (78.7 vs 78.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use direct speech input instead of an external ASR pipeline | VLAS versus VLA + ASR on CALVIN ABCD/D · CALVIN ABCD/D, speech instructions | Len: 3.13 -> 3.70 | Helped |
| Add Voice RAG (ablation) | VLAS · Customization benchmark | Avg. success rate: 16.0% -> 86.5% | Helped |
| Add Voice RAG | VLA · Customization benchmark | Avg. success rate: 19.2% -> 82.0% | Helped |
| Add speech-input capability | LLaVA v1.5 to form VLAS-Base · VQA v2 | score: 78.8 -> 78.7 | No clear effect |
| Add speech-input capability | LLaVA v1.5 to form VLAS-Base · SGQA | accuracy: 62.0 -> 50.8 | Hurt |
| Use r=5 rather than r=1 multi-step prediction and execution (ablation) | VLAS · CALVIN | Len: 2.02 -> 3.70 | Helped |
| Increase action prediction and execution steps from r=5 to r=20 (ablation) | VLAS · CALVIN | Len: 3.70 -> 0.70 | Hurt |
| Evaluate with real recorded speech instead of generated speech | VLAS · CALVIN ABCD/D | Len: 3.70 -> 3.61 | Hurt |
Surprising findings
- On CALVIN ABC/D generalization splits, VLAS with speech and the text-only VLA perform similarly in Len (2.40 vs 2.14), with VLAS performing better in the speech setting.
- The VLA baseline with RAG reaches 82.0% average on the customized benchmark, comparable to VLAS, indicating the gain comes largely from retrieved knowledge.
Limitations
- Performance on the customized benchmark is evaluated in simulation with 39 unseen TTS voices, with a smaller real-speech evaluation.
- VLAS trails the text-only VLA on CALVIN with textual instructions (Len 3.74 vs 3.80) and RoboFlamingo with history.
- Failure cases in preference and second-stage compound tasks indicate execution errors remain.
Experimental setup
Datasets: SQA CSI LLaVA 665K instruction data LibriSpeech CALVIN GQA Berkeley UR5 demonstration dataset LibriTTS
Benchmarks: CALVIN customized manipulation benchmark (ownership, preference, compound tasks) VQA-v2 VizWiz SQA-I VQA-T POPE GQA LibriSpeech SGQA
Metrics: success rate per long-horizon step (LH-1 to LH-5) average task length (Len) success rate word error rate accuracy
Baselines: MCIL HULC RT-1 VLA (LLaVA-based, text-only) Roboflamingo OpenVLA Whisper large-v2 + ASR cascade BLIP-2
Builds on: LLaVA Whisper RT-2 RoboFlamingo Retrieval-Augmented Generation
Closest papers by meaning
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (ICLR 2025 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.89)
- ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuning (NeurIPS 2026 · similarity 0.89)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.89)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.