Research map / Vision-language and multimodal / Robot action and navigation
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeeR-VLA, a multi-exit OpenFlamingo-based robot policy that dynamically picks how many LLM layers to run per timestep via action-consistency exit criteria, cutting LLM FLOPs 5-6x on CALVIN.
Problem
Running billion-parameter MLLMs for every robot action is computationally expensive and memory-hungry, which hinders deployment of generalist MLLM-based robot policies on resource-limited platforms.
Approach
Equip the MLLM LLM with intermediate exits and an LSTM action head, terminate at the first exit whose predicted action is consistent with the previous exit's action (L2 below a threshold), with thresholds set by a demo-data quantile scheme or Bayesian optimization under FLOPs, peak FLOPs and memory budgets; train with random exit sampling plus auxiliary per-exit action heads.
Key results
- On CALVIN D→D, DeeR reaches 2.83 average successful length at 8.6 GFLOPs versus 2.71 for RoboFlamingo++ at 31.2 GFLOPs.
- DeeR reduces LLM computation by 5.2-6.5x and LLM GPU memory by 2-6x without performance loss on CALVIN.
- With OpenFlamingo 9B, DeeR reduces computation 1.8-5.7x and peak FLOPs and memory 2.7-4.0x for the same performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic early exits instead of fixed full-model inference | RoboFlamingo++ on OpenFlamingo 3B · CALVIN D→D | average successful length: 2.71 -> 2.83 | Helped |
| use dynamic early exits instead of fixed full-model inference | RoboFlamingo++ on OpenFlamingo 3B · CALVIN ABCD→D | average successful length: 4.07 -> 4.13 | Helped |
| use dynamic early exits instead of fixed full-model inference | RoboFlamingo++ on OpenFlamingo 3B · CALVIN ABC→D | average successful length: 2.59 -> 2.82 | Helped |
| solve exit thresholds with online interaction | DeeR · CALVIN D→D | average successful length: 2.83 -> 2.92 | Helped |
| solve exit thresholds with online interaction | DeeR · CALVIN ABCD→D | average successful length: 4.13 -> 4.13 | No clear effect |
| solve exit thresholds with online interaction | DeeR · CALVIN ABC→D | average successful length: 2.82 -> 2.90 | Helped |
| add auxiliary losses at intermediate exits (ablation) | DeeR · CALVIN ABCD→D, 4.9 GFLOPs | average successful length: 2.64 -> 3.94 | Helped |
| add auxiliary losses at intermediate exits (ablation) | DeeR · CALVIN ABCD→D, 10.0 GFLOPs | average successful length: 2.71 -> 4.13 | Helped |
| use action-consistency exits instead of feature-similarity exits (ablation) | DeeR · CALVIN D→D, 4.9 GFLOPs | average successful length: 2.52 -> 2.65 | Helped |
| use action-consistency exits instead of time-based exits (ablation) | DeeR · CALVIN ABC→D, 4.9 GFLOPs | average successful length: 2.46 -> 2.62 | Helped |
| quantize DeeR to float16 (ablation) | DeeR · CALVIN ABCD→D | memory: 6G -> 3G | Helped |
| quantize DeeR to float16 (ablation) | DeeR · CALVIN ABCD→D | average successful length: 4.13 -> 4.12 | No clear effect |
Surprising findings
- DeeR-S achieves high performance with only about 2GB of LLM GPU memory.
- Confidence-style exit criteria (softmax confidence, entropy) are infeasible here because actions are regressed, so an action-consistency criterion is used instead.
Limitations
- Only evaluated in the simulated CALVIN benchmark, not on real robots.
- Visual encoder compute is not reduced and remains significant.
- Early-exit implementation not code-optimized, so wall-clock gains are below FLOPs reduction.
Experimental setup
Datasets: CALVIN
Benchmarks: CALVIN LH-MTLC
Metrics: average successful length GFLOPs GPU memory
Baselines: RoboFlamingo RoboFlamingo++ GR-1 HULC SPIL SuSIE RT-1
Builds on: RoboFlamingo++ RoboFlamingo OpenFlamingo
Closest papers by meaning
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.89)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.89)
- LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (ICLR 2025 · similarity 0.89)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.88)
- VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.