Research map / Multi-agent systems and games / Human-AI and goal-conditioned
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the LLaVA-OneVision MLLM into GEA, a single generalist embodied agent that acts via a multi-embodiment action tokenizer, trained with SFT on 2.2M trajectories and then online PPO RL in three simulators.
Problem
Building a single multimodal LLM-based agent that can control diverse embodiments (manipulation, navigation, video games, UI control, planning) and generalize to unseen tasks, where SFT alone suffers from limited data diversity and poor robustness to mistakes.
Approach
A unified learned residual VQ-VAE action tokenizer across all continuous action spaces, embodiment-prompted MLLM SFT on a large multi-domain embodied dataset, followed by a second stage of online PPO RL (with LoRA on the LLM and SFT loss mixed in) on a subset of domains.
Key results
- GEA reaches 90% success on CALVIN (ABC→D), nearly 10% above MLLM+IL baseline (82.4%) and close to 3D Diffuser Actor (92.2%).
- Online PPO RL raises GEA-Base success on Habitat Pick from 57% to 83% (LLaVA-OV-500m ablation).
- GEA achieves 44.0% of expert score on Procgen versus 25% for the prior specialist, and 94.7% on Meta-World versus 87.0% for prior generalist.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage 2 online RL to SFT training (ablation) | GEA-Base · Habitat Pick | success rate (%): 60.5 -> 82.5 | Helped |
| Add Stage 2 online RL to SFT training (ablation) | GEA-Base · Procgen | expert performance (%): 36.1 -> 44.0 | Helped |
| Add Stage 2 online RL to SFT training (ablation) | GEA-Base · LangR | success rate (%): 15.5 -> 50.0 | Helped |
| Add Stage 2 online RL to SFT training (ablation) | GEA-Base · All Other tasks | success rate (%): 69.5 -> 70.5 | No clear effect |
| Train with combined cross-domain data instead of only target-domain data (ablation) | GEA-Base · Habitat Pick | success rate (%): 54.5 -> 57.0 | Helped |
| Train with combined cross-domain data instead of only target-domain data (ablation) | GEA-Base · CALVIN | success rate (%): 35.5 -> 48.0 | Helped |
| Train with combined cross-domain data instead of only target-domain data (ablation) | GEA-Base · Procgen | expert performance (%): 23.7 -> 24.5 | Helped |
| Train with combined cross-domain data instead of only target-domain data (ablation) | GEA-Base · AndroidControl | success rate (%): 48.9 -> 50.5 | Helped |
| Train with combined cross-domain data instead of only target-domain data (ablation) | GEA-Base · BabyAI | success rate (%): 82.1 -> 84.7 | Helped |
| Add online PPO fine-tuning rather than successful-example SFT or offline IQL (ablation) | GEA-Base · Habitat Pick | success rate: 57% to 83% | Helped |
| Add online PPO fine-tuning rather than offline IQL (ablation) | GEA-Base · Habitat Pick | success rate | Helped |
| Initialize the full model from a pretrained MLLM rather than initialize only the LLM subnet (ablation) | GEA-Base · Habitat Pick | success rate (%): 9.5 -> 57.0 | Helped |
Surprising findings
- Visual encoder initialization matters more than LLM initialization for final performance, as training the SigLIP encoder from scratch on embodied data is hard.
- The class of base MLLM has little effect; three 7B models from different families performed nearly identically.
Limitations
- Cannot control arbitrary embodiments or operate in arbitrary environments zero-shot.
- Performance remains far from perfect on Maniskill, Atari, and AndroidControl.
- Context limited to three previous observations, which may hurt partially observable tasks such as Habitat Nav.
Experimental setup
Datasets: OpenX Meta-World CALVIN Maniskill Habitat Pick Habitat Place Habitat Nav BabyAI LangR Procgen Atari AndroidControl
Benchmarks: Meta-World CALVIN Maniskill Habitat Pick Habitat Place Procgen Atari Habitat Nav BabyAI AndroidControl LangR
Metrics: success rate percent of expert score (video games)
Baselines: Gato 3D Diffuser Actor Multi-Game DT GPT-4o with set-of-marks MLLM+IL RL + sim state Maniskill IL+PPO OpenVLA
Builds on: LLaVA-OneVision Szot et al. multi-embodiment action tokenizer (MLLM-as-agent) PPO Gato
Closest papers by meaning
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.91)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.90)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.90)
- Large Language Models as Generalizable Policies for Embodied Tasks (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.