Research map / Vision-language and multimodal / Robot action and navigation
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a vision-language agent (EMMA) in the visual ALFWorld by imitating a retrospective ChatGPT-style LLM expert running in the parallel text world, using a DAgger-DPO objective.
Problem
VLM agents lack alignment with the dynamics of embodied visual worlds, and training embodied agents in noisy visual environments without expert guidance is inefficient; sparse rewards and hallucination make online RL hard.
Approach
Distill an LLM expert's reflection-improved actions from a parallel text world into a VLM agent in the visual world via interactive imitation learning (DAgger) with a DPO loss, where the LLM expert is an actor plus a critic that writes retrospective feedback stored in long-term memory.
Key results
- EMMA reaches 0.82 average success rate on ALFWorld visual environment (134 OOD tasks, template instructions), vs 0.22 for InstructBLIP and 0.06-0.16 for other VLMs; GPT-4V is not reported in the table (Fig. 1 only).
- EMMA reports 0.67 average success on free-form human-annotated instructions vs near-zero for other VLM agents (e.g., InstructBLIP 0.01).
- Ablations show retrospection improves EMMA, and DPO outperforms cross-entropy loss; EMMA is more robust to noisy observations than Reflexion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the VLM with cross-modality interactive imitation learning from an LLM expert | InstructBLIP on ALFWorld visual tasks · ALFWorld OOD tasks, template instructions, visual environment | success rate: 0.22 -> 0.82 | Helped |
| Train with retrospective LLM feedback rather than using only a plain LLM actor (ablation) | EMMA · ALFWorld visual tasks across training trials | success rate | Helped |
| Remove behavior-cloning initialization (ablation) | EMMA · 134 unseen ALFWorld tasks | success rate | Hurt |
| Replace DPO loss with token-level cross-entropy loss (ablation) | EMMA · ALFWorld visual tasks | success rate | Hurt |
| Replace DPO loss with token-level cross-entropy loss (ablation) | EMMA · ALFWorld visual tasks during initial training | convergence speed | Helped |
| Use visual observations rather than textual observations under increasing observation noise | EMMA versus Reflexion · ALFWorld tasks with noisy observations | success rate | Helped |
| Use free-form human-annotated instructions instead of template instructions | EMMA · 134 unseen ALFWorld tasks | success rate | Hurt |
Surprising findings
- EMMA is more robust to noisy observations than the LLM agent Reflexion, attributed to the vision encoder filtering visual noise while textual noise directly affects LLM processing.
- Behavior cloning initialization contributes only slightly; EMMA without BC init still outperforms other VLM agents.
Limitations
- Relies on a closed API LLM (text-davinci-003) as expert, and the approach needs a textual simulator equivalent of visual states.
- Slight performance drop on free-form human-annotated instructions compared with template instructions.
- Evaluated only on ALFWorld benchmark.
Experimental setup
Datasets: ALFWorld ALFWorld rule-based expert demonstrations (170K)
Benchmarks: ALFWorld
Metrics: success rate average interaction steps
Baselines: GPT-4V InstructBLIP MiniGPT-4 BLIP-2 LLaMA-Adapter Reflexion ReAct GPT-BUTLER BUTLER DEPS AutoGen ResNet-18
Builds on: DAgger DPO Reflexion InstructBLIP ALFWorld
Closest papers by meaning
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.90)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.89)
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.89)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.