Research map / Vision-language and multimodal / Robot action and navigation
Vision-Language Foundation Models as Effective Robot Imitators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the open-source OpenFlamingo VLM into a robot manipulation policy by adding an LSTM policy head and fine-tuning only the cross-attention, resampler and head via imitation learning on CALVIN.
Problem
Adapting pre-trained vision-language models for low-level, closed-loop language-conditioned robot manipulation, which is hard because VLMs are trained on static image-text pairs and output text rather than actions, and prior approaches need private models or huge co-training data.
Approach
Decouple single-step vision-language understanding (a Flamingo/OpenFlamingo backbone) from history modeling (an explicit LSTM policy head over max-pooled fused tokens), and fine-tune only the resampler, gated cross-attention and policy head with imitation learning on language-annotated manipulation data.
Key results
- On CALVIN ABCD->D, RoboFlamingo (MPT-3B-IFT) reaches 4.09 average length versus 3.06 for HULC (ABCD full data).
- On ABC->D, RoboFlamingo achieves 2.48 average length versus 0.67 for HULC with full data.
- With only 10% of language-annotated data, the MPT-9B variant reaches 0.83 average length, showing larger VLMs are more data-efficient.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoboFlamingo instead of HULC | language-conditioned manipulation policy · CALVIN ABCD→D | Avg Len: 3.06 -> 4.09 | Helped |
| Use RoboFlamingo instead of RT-1 | language-conditioned manipulation policy · CALVIN ABC→D | Avg Len: 0.90 -> 2.48 | Helped |
| freeze the fusion decoder embedding layer (ablation) | RoboFlamingo · CALVIN ABCD→D (Enriched) | Avg Len: 1.85 -> 2.12 | Helped |
| instruction fine-tune the M-3B backbone (ablation) | RoboFlamingo with M-3B · CALVIN ABCD→D (10% language data) | Avg Len: 0.05 -> 0.13 | Helped |
| instruction fine-tune the G-4B backbone (ablation) | RoboFlamingo with G-4B · CALVIN ABCD→D (10% language data) | Avg Len: 0.48 -> 0.55 | Helped |
| increase backbone size from M-3B to M-9B (ablation) | RoboFlamingo · CALVIN ABCD→D (10% language data) | Avg Len: 0.05 -> 0.83 | Helped |
| fine-tune Voltron representations instead of freezing them (ablation) | Voltron policy · CALVIN ABC→D | Avg Len: 0.03 -> 1.00 | Helped |
| co-train with COCO and VQA data instead of robotics-only fine-tuning (ablation) | RoboFlamingo M-3B-IFT · CALVIN ABCD→D | Avg Len: 4.09 -> 3.76 | Hurt |
| fine-tune all model parameters instead of only the resampler, gated cross-attention, and policy head (ablation) | RoboFlamingo M-3B-IFT · CALVIN ABCD→D | Avg Len: 4.09 -> 0.50 | Hurt |
| use a single-step MLP without observation history instead of a history-aware policy head (ablation) | RoboFlamingo · CALVIN ABCD→D | Avg Len | Hurt |
| use MLP with history in the fusion decoder instead of GPT or LSTM policy heads (ablation) | RoboFlamingo · CALVIN ABCD→D | Avg Len | Hurt |
| remove vision-language pre-training from the resampler and cross-attention layers (ablation) | RoboFlamingo · CALVIN ABCD→D | Avg Len | Hurt |
Surprising findings
- Full-model fine-tuning (3B trainable) degrades performance sharply (0.50 average length) compared with the frozen-backbone variant (4.09).
- Open-loop control without retraining deteriorates performance, though retraining with jump-step demonstrations alleviates the drop.
Limitations
- No real-robot experiments; evaluated only in simulation on CALVIN.
- Performance drops on GPT-4-enriched synonymous instructions and some VL abilities degrade after fine-tuning.
Experimental setup
Datasets: CALVIN COCO VQAv2
Benchmarks: CALVIN
Metrics: success rate per task in a 5-task sequence average sequence length
Baselines: MCIL HULC RT-1 Voltron R3M
Builds on: Flamingo OpenFlamingo RT-1 HULC
Closest papers by meaning
- RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation (NeurIPS 2024 · similarity 0.90)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.89)
- VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (NeurIPS 2024 · similarity 0.89)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.