Research map / Vision-language and multimodal / Robot action and navigation
Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes knowledge insulation for VLAs: train the VLM backbone on discretized FAST action tokens and VLM data while a flow-matching action expert is trained with stop-gradient into the backbone, yielding faster training and better language following than pi0.
Problem
Continuous-action VLAs that graft a randomly initialized diffusion/flow-matching action expert onto a pretrained VLM train slowly and degrade the VLM's semantic knowledge and language following.
Approach
Jointly train the VLM backbone with next-token prediction on FAST discretized actions (plus VLM data) and a flow-matching action expert, while stopping gradients from the action expert into the backbone via a modified attention with stop-gradient on backbone keys/values; FAST tokens are invisible to continuous action tokens.
Key results
- On DROID, the method scored 0.55 ± 0.09 versus pi0 at 0.49 ± 0.09 and pi0-FAST at 0.45 ± 0.09.
- Training converges as fast as pi0-FAST, while pi0 needs about 7.5x as many training steps to reach similar table-bussing performance.
- Achieves new state-of-the-art on LIBERO-90 and LIBERO-Spatial per the paper's comparison table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use knowledge insulation with discrete action training and a continuous action expert | VLA · DROID tabletop manipulation | average score: 0.49 ± 0.09 -> 0.55 ± 0.09 | Helped |
| Add stop-gradient from the action expert to the backbone (ablation) | Joint-training VLA · Language following | language following | Helped |
| Remove stop-gradient from the action expert to the backbone (ablation) | Knowledge insulation VLA · Items in drawer | task performance | Hurt |
| Remove co-training on VLM data (ablation) | Generalist VLA · Table bussing | task completion | Hurt |
| Add co-training on VLM data (ablation) | Joint-training VLA · Language following | language following | Helped |
| Replace FAST action tokens with dense naive tokenization (ablation) | Knowledge insulation VLA · Table bussing | task performance | Hurt |
| Subsample naive action tokens with stride 5 instead of using dense naive tokenization (ablation) | Knowledge insulation VLA · Table bussing | task performance | Helped |
| Freeze the pretrained backbone instead of fine-tuning it (ablation) | VLA · Robotic manipulation | task performance | Hurt |
| Replace text or continuous state with special-token state (ablation) | Knowledge insulation VLA · Table bussing | task performance | Hurt |
| Replace FAST with naive action tokenization (ablation) | Knowledge insulation VLA · Table bussing | task performance | Mixed |
| Train with flow-matching loss only instead of adding FAST action-token representation learning (ablation) | VLA · Training convergence | training steps to convergence: 7.5 times as many training steps | Hurt |
| Allow autoregressive action tokens to attend to continuous action inputs (ablation) | HybridVLA · Items in drawer | task performance | Hurt |
Surprising findings
- Naively adding a randomly initialized flow-matching action expert degrades language following and slows training relative to the VLM-based autoregressive approach.
- Freezing the VLM backbone yields near-zero performance on some tasks, showing pretrained VLM representations are insufficient for robotics without adaptation.
Limitations
- Joint discrete and continuous training increases per-step training cost by about 20%.
- Language following remains imperfect because training correlations still cause the model to sometimes ignore instructions.
- Most real-world evaluation uses proprietary data and robots, limiting reproducibility.
Experimental setup
Datasets: LIBERO DROID OXE CapsFusion COCO Cambrian-7M PixMo VQAv2 proprietary robot data from multiple embodiments
Benchmarks: LIBERO-90 LIBERO-Spatial LIBERO-Object LIBERO-Goal LIBERO-Long DROID real-world table bussing items in drawer shirt folding mobile manipulation tasks
Metrics: task score (points out of max) language following rate success rate inference time training steps to convergence
Baselines: pi0 pi0-FAST OpenVLA-OFT Transfusion HybridVLA joint-training
Builds on: pi0 pi0-FAST PaliGemma FAST action tokenizer HybridVLA
Closest papers by meaning
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.