Research map / Vision-language and multimodal / Robot action and navigation
VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares VLA task-planning paradigms (ActionOnly, Integrated, Hierarchical) and planning representations (language, visual, image foresight) using a unified VLA-OS model family under controlled experiments.
Problem
Existing VLA task-planning approaches differ in architecture, planning paradigm, representation, and training data, so it is unclear which design choices drive performance gains in long-horizon manipulation.
Approach
A unified, composable VLA model family (VLA-OS-A/I/H) with Qwen2.5 backbones and plug-and-play planning heads (language, visual location tokens, image foresight via VAR-style generation) enables controlled comparison of paradigms and representations with architecture and data held fixed.
Key results
- Visual planning and image foresight outperform language planning: VLA-OS-I-I improves by +5.75 (V) and +6.5 (IF) over ActionOnly-VLA vs +2.0 for language on LIBERO-Long.
- VLA-OS-H with all three representations reaches 74.2% on LIBERO-Long, versus 72.0% for language-only-free baseline comparisons reported in the table (VLA-OS-A reference 66.0).
- VLA-OS-A-S (0.5B, scratch) averages 85.6% on four LIBERO suites, vs pi0-FAST 85.5% and pi0 94.2% (SOTA, fine-tuned).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use implicit rather than explicit planning (ablation) | Integrated-VLA on LIBERO-Long · LIBERO-Long | success rate: 68.0 vs 60.5 (L); 71.0 vs 52.5 (V); 72.5 vs 67.5 (IF) | Helped |
| use visual planning rather than language planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 71.0 | Helped |
| use image foresight planning rather than language planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 72.5 | Helped |
| combine language and visual planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 66.7 | Hurt |
| combine language and image foresight planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 73.3 | Helped |
| combine visual and image foresight planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 71.0 | Helped |
| combine language, visual, and image foresight planning (ablation) | VLA-OS-I-I on LIBERO-Long · LIBERO-Long | success rate: 68.0 -> 71.7 | Helped |
| use visual planning rather than language planning (ablation) | VLA-OS-H on LIBERO-Long · LIBERO-Long | success rate: 63.5 -> 69.0 | Helped |
| use image foresight planning rather than language planning (ablation) | VLA-OS-H on LIBERO-Long · LIBERO-Long | success rate: 63.5 -> 71.7 | Helped |
| combine language, visual, and image foresight planning (ablation) | VLA-OS-H on LIBERO-Long · LIBERO-Long | success rate: 63.5 -> 74.2 | Helped |
| use Hierarchical-VLA rather than Integrated-VLA for task planning | VLA-OS on LIBERO-Long · LIBERO-Long | Decomposition Score: 0.79 -> 0.81 (L); 0.83 -> 0.86 (V); 0.92 -> 0.94 (IF) | Helped |
| use visually grounded rather than language-based planning | VLA-OS-H low-level policy on LIBERO-Long · LIBERO-Long | Instruction Following Score: 0.84 -> 0.93 (V); 0.84 -> 0.90 (IF) | Helped |
Surprising findings
- Larger VLA models trained on the data do not necessarily outperform smaller models trained from scratch; the 0.5B model is competitive.
- Explicit planning (chain-of-thought style) in Integrated-VLA degrades performance significantly when trained from scratch.
Limitations
- Training dataset is limited to fewer than 10,000 trajectories; pretraining on larger datasets like OXE not explored.
- Embodiment transfer, sim2real, and 2D-to-3D transfer were not investigated.
- Hierarchical-VLA incurs higher training cost and slower inference, and shows faster forgetting in continual learning than ActionOnly.
Experimental setup
Datasets: LIBERO LIBERO-90 LLaVA v1.5 data mixture COLOSSEUM FurnitureBench DexArt PerAct2 real-world deformable manipulation dataset real-world rigid-body manipulation dataset
Benchmarks: LIBERO LIBERO-Long COLOSSEUM FurnitureBench DexArt PerAct2 real-world deformable tasks real-world rigid-body tasks
Metrics: success rate decomposition score (DCS) instruction following score (IFS) forward transfer (FWT) negative backward transfer (NBT) AUC
Baselines: Diffusion Policy OpenVLA CoT-VLA DiT Policy pi0 pi0-FAST
Builds on: pi0 VAR 3D Diffuser Actor Embodied Chain-of-Thought Qwen2.5
Closest papers by meaning
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.92)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.91)
- Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.