Research map / Vision-language and multimodal / Robot action and navigation
Grounding Multimodal Large Language Models in Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of seven action space adapters for grounding LLaVA-1.5-7B into continuous and discrete embodied action spaces across five environments, finding learned residual VQ tokenization best for continuous control and semantic language tokenization best for discrete actions.
Problem
How best to ground a multimodal LLM into embodied action spaces, which differ in type (continuous vs discrete) and sit far from the MLLM's natural language output space.
Approach
Unifies prior MLLM-to-action methods as Action Space Adapters (ASAs) and systematically compares them; proposes residual VQ tokenization for continuous actions and semantically aligned language tokens (SemLang) for discrete actions.
Key results
- On Meta-World ML-45, RVQ tokenization achieves 84% success vs 61% for direct regression (Pred) and 75% for uniform tokenization.
- On CALVIN ABC->D, RVQ reaches 72% success vs 68% for Pred and 28% for Uniform.
- On Language Rearrangement, SemLang reaches 51% success vs 42% for Pred, and outperforms Lang by 14 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RVQ instead of direct action regression | LLaVA-1.5-7B continuous-action policy · CALVIN ABC → D | success rate: 68 → 72 | Helped |
| Use RVQ instead of direct action regression | LLaVA-1.5-7B continuous-action policy · Meta-World ML-45 | success rate: 61 → 84 | Helped |
| Use RVQ instead of uniform action tokenization | LLaVA-1.5-7B continuous-action policy · CALVIN ABC → D | success rate: 28 → 72 | Helped |
| Use RVQ instead of single-codebook VQ | LLaVA-1.5-7B continuous-action policy · Meta-World ML-45 | reconstruction MSE: 0.05 → 0.005 | Helped |
| Reduce RVQ codebook size from 512 to 16 (ablation) | LLaVA-1.5-7B RVQ policy · Meta-World ML-45 | success rate: 84\% → 29\% | Hurt |
| Increase RVQ number of codebooks to 6 (ablation) | LLaVA-1.5-7B RVQ policy · Meta-World ML-45 | success rate | Hurt |
| Use RVQ instead of direct action regression (ablation) | Scratch 300M policy · CALVIN ABC → D | success rate: 50 → 43 | Hurt |
| Use RVQ instead of direct action regression (ablation) | Scratch 300M policy · Meta-World ML-45 | success rate: 71 → 56 | Hurt |
| Use RVQ instead of direct action regression (ablation) | Scratch 300M policy · Habitat Pick | success rate: 21 → 25 | Helped |
| Use RVQ instead of direct action regression (ablation) | RT-Inspired policy · Meta-World ML-45 | success rate: 27 → 38 | Helped |
| Use SemLang instead of categorical action prediction | LLaVA-1.5-7B discrete-action policy · Language Rearrangement | success rate: 42 { $ \pm$ 2 } → 51 { $ \pm$ 1 } | Helped |
| Use SemLang instead of non-semantic digit tokens | LLaVA-1.5-7B discrete-action policy · Language Rearrangement | success rate: 27 { $ \pm$ 12 } → 51 { $ \pm$ 1 } | Helped |
Surprising findings
- RVQ tokenization hurts performance relative to direct regression when the policy is trained from scratch (Scratch: -7 on CALVIN, -15 on Meta-World), while it helps the MLLM-based policy, suggesting the LLM leverages its sequencing knowledge.
- Increasing VQ codebook size beyond 2^9 decreases VQ policy performance on Meta-World, possibly due to overfitting.
Limitations
- Analysis limited to a single MLLM (LLaVA-1.5) and a single LoRA training setting.
- RVQ requires collected demonstrations to train the VQ model.
- Performance remains low on simple tasks (e.g., BabyAI 40%), limiting real-robot deployment.
Experimental setup
Datasets: CALVIN ABC->D Meta-World ML-45 Habitat Pick (Habitat 2.0) BabyAI Language Rearrangement
Benchmarks: CALVIN ABC->D Meta-World ML-45 Habitat Pick BabyAI Language Rearrangement
Metrics: success rate
Baselines: Pred Uniform VQ Lang Scratch RT-Inspired RoboFlamingo LLaRP GFlan DualMind
Builds on: LLaVA-1.5 VQ-VAE Residual VQ-VAE RT-2 LoRA PPO
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- Grounding Everything in Tokens for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.89)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.89)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.89)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.