Research map / Vision-language and multimodal / Robot action and navigation
CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CogVLA, a cognition-inspired three-stage VLA framework (instruction-conditioned FiLM vision aggregation, LLM-side token pruning, and a hybrid causal/bidirectional attention with parallel action decoding) that reaches 97.4% on LIBERO with lower training and inference cost than OpenVLA.
Problem
Fine-tuning VLM-based VLA models for action prediction is computationally expensive, and existing sparsification methods (Mixture-of-Depths, layer skipping, early exit) optimize within the LLM only, causing cross-modal semantic degradation.
Approach
Instruction-driven sparsification across the whole perception-to-control pipeline: FiLM-conditioned aggregation inside the vision encoders (EFA-Routing), FiLM-conditioned token pruning inside the LLM (LFP-Routing), and a V-L-A coupled attention mask that is causal over vision-language and bidirectional over action chunks (CAtten).
Key results
- CogVLA reaches 97.4% average success on LIBERO (98.6 Spatial, 98.8 Object, 96.6 Goal, 95.4 Long), the best average among compared methods.
- Compared to OpenVLA, CogVLA reports 2.79x faster inference (0.091 s vs 0.254 s), 3.12x lower FLOPs (2.72T vs 8.48T), and 2.49x lower training cost (4.7 vs 11.7 h per 10k steps).
- On real-world ALOHA tasks CogVLA achieves 70.0% average success versus 56.7% for OpenVLA-OFT (reproduced).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CogVLA rather than OpenVLA-OFT | Vision-Language-Action models · LIBERO Average | success rate: 97.1 -> 97.4 | Helped |
| use CogVLA rather than OpenVLA-OFT | Vision-Language-Action models · ALOHA Tasks 1–3 | success rate: 56.7% -> 70.0% | Helped |
| add Stage 1 (EFA-Routing) (ablation) | CogVLA · LIBERO | inference time: 0.162 s -> 0.091 s | Helped |
| add Stage 2 (LFP-Routing) (ablation) | CogVLA · LIBERO | inference time: 0.117 s -> 0.091 s | Helped |
| add Stage 1 (EFA-Routing) (ablation) | CogVLA · LIBERO Spatial | success rate: 91.2 -> 98.6 | Helped |
| add Stage 1 Step 1 (ablation) | CogVLA · LIBERO Spatial | success rate: 96.0 -> 98.6 | Helped |
| add Stage 1 Step 2 (ablation) | CogVLA · LIBERO Spatial | success rate: 95.2 -> 98.6 | Helped |
| add LFP-Routing pruning (ablation) | CogVLA · LIBERO Spatial | success rate: 92.0 -> 98.6 | Helped |
| add instruction guidance to pruning (ablation) | CogVLA · LIBERO Spatial | success rate: 96.2 -> 98.6 | Helped |
| add CAtten (ablation) | CogVLA · LIBERO Spatial | success rate: 92.0 -> 98.6 | Helped |
| allocate sparsification as Stage 1 4× and Stage 2 2× rather than Stage 1 2× and Stage 2 4× (ablation) | CogVLA · LIBERO Spatial | success rate: 94.6 -> 98.6 | Helped |
| increase both stage sparsification ratios from 2×–2× to 4×–4× (ablation) | CogVLA · LIBERO Spatial | success rate: 96.4 -> 93.2 | Hurt |
Surprising findings
- Asymmetric sparsity allocation (more aggressive Stage 1 compression) outperforms symmetric allocation, with 4x Stage 1 and 2x Stage 2 giving the best Spatial SR.
Limitations
- Sparsity ratios and pruning schedules are predefined and fixed rather than adaptive to instruction complexity or scene difficulty.
- Generalization to out-of-distribution instructions or unseen manipulation categories is not thoroughly evaluated.
- Real-world evaluation uses limited trials (10 per subtask) and remote-communication latency affects demo videos.
Experimental setup
Datasets: LIBERO real-world Cobot Agilex ALOHA demonstrations
Benchmarks: LIBERO real-world Cobot Agilex ALOHA tasks
Metrics: success rate inference time throughput FLOPs training cost
Baselines: Diffusion Policy Octo OpenVLA pi0 pi0-Fast pi0.5-KI OpenVLA-OFT SpatialVLA PD-VLA STAR Dita CoT-VLA
Builds on: OpenVLA FiLM OpenVLA-OFT FastV
Closest papers by meaning
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.93)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- CofactVLA: Deconfounding Vision-Language-Action Models via Counterfactual Intervention (NeurIPS 2026 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.