Research map / Vision-language and multimodal / Robot action and navigation
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoT-VLA, a 7B vision-language-action model that first autoregressively generates a subgoal image as visual chain-of-thought and then predicts an action chunk, pretrained on Open X-Embodiment robot data plus action-less videos.
Problem
Current vision-language-action models map observations and instructions directly to actions with no intermediate reasoning or temporal planning, and cannot exploit action-less video data for visual reasoning.
Approach
Use predicted future subgoal images as an intermediate visual chain-of-thought step before action generation, built on a unified autoregressive image-text model (VILA-U), with causal attention for image generation and full attention for parallel action-chunk decoding, so action-less videos can train the visual reasoning step.
Key results
- CoT-VLA-7B achieves 81.13% average success on LIBERO, above OpenVLA fine-tuned at 76.5% and Octo at 75.1%.
- On Franka-Tabletop, pretraining raises success from 53.7% to 78.8%, a 46.7% relative improvement over fine-tuning VILA-U directly.
- Using ground-truth goal images instead of generated ones improves success on out-of-distribution long-horizon tasks by 40 points on both sub-tasks (e.g., 20% to 60% on sub-task 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual chain-of-thought subgoal image generation (ablation) | CoT-VLA on LIBERO-Spatial and LIBERO-Goal · LIBERO-Spatial and LIBERO-Goal | Success rate | Helped |
| add action chunking (predict action sequences) (ablation) | VLA on LIBERO-Spatial and LIBERO-Goal · LIBERO-Spatial and LIBERO-Goal | Success rate | Helped |
| add hybrid full attention for action sequence prediction (ablation) | action-chunked VLA on LIBERO-Spatial and LIBERO-Goal · LIBERO-Spatial and LIBERO-Goal | Success rate | Helped |
| pretrain VILA-U on OpenX and action-less videos before Franka-Tabletop fine-tuning (ablation) | CoT-VLA on Franka-Tabletop · Franka-Tabletop | Success rate: 53.7% to 78.8%; 46.7% relative improvement | Helped |
| use ground-truth rather than generated goal images (ablation) | CoT-VLA on Franka-Tabletop out-of-distribution tasks · Sub-task 1 | Success rate: 20% -> 60% | Helped |
| use ground-truth rather than generated goal images (ablation) | CoT-VLA on Franka-Tabletop out-of-distribution tasks · Sub-task 2 | Success rate: 0% -> 40% | Helped |
| use visual chain-of-thought CoT-VLA instead of OpenVLA | VLA models on Bridge-V2 · Bridge-V2 visual generalization | Success rate: 75% -> 65% | Hurt |
| use visual chain-of-thought CoT-VLA instead of OpenVLA | VLA models on Bridge-V2 · Bridge-V2 motion generalization | Success rate: 45% -> 60% | Helped |
| use visual chain-of-thought CoT-VLA instead of OpenVLA | VLA models on Bridge-V2 · Bridge-V2 semantic generalization | Success rate: 40% -> 50% | Helped |
| use visual chain-of-thought CoT-VLA instead of OpenVLA | VLA models on Bridge-V2 · Bridge-V2 language grounding | Success rate: 75% -> 70% | Hurt |
Surprising findings
- Generating 256 image tokens before actions causes about a 7x inference slowdown with action chunk size 10.
- On Bridge-V2, CoT-VLA is slightly below OpenVLA on visual and language generalization tasks, attributed to grasping failures from action chunking.
Limitations
- Image token generation adds significant inference overhead (about 7x slowdown).
- Autoregressive image generation yields lower visual quality than diffusion-based models such as SUSIE.
- Out-of-distribution subgoal generation remains weak, limiting generalization to entirely new tasks.
Experimental setup
Datasets: Open X-Embodiment Bridge-V2 EPIC-KITCHENS Something-Something V2 LIBERO
Benchmarks: LIBERO Bridge-V2 Franka-Tabletop
Metrics: success rate
Baselines: Diffusion Policy Octo OpenVLA SUSIE
Builds on: VILA-U OpenVLA SUSIE RQ-VAE
Closest papers by meaning
- VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching (NeurIPS 2025 · similarity 0.92)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.91)
- BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.