Research map / Large language models / Language, safety and interpretability
Large Language Models as Generalizable Policies for Embodied Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaRP, which adapts a frozen pretrained LLM (LLaMA-7B) with trained observation encoder and action head via PPO RL to act from egocentric vision and language instructions, and releases the Language Rearrangement benchmark.
Problem
Generalizable language-conditioned embodied policies that take visual observations and text instructions and act directly in the environment; existing LLM-for-embodied work relies on static expert data or zero-shot planning that is blind to vision.
Approach
Use a pre-trained frozen LLM backbone (input/output layers stripped) with learnable visual encoder MLP and action output MLP, trained only by online PPO RL with sparse/subgoal rewards, so the LLM's world knowledge transfers to unseen instructions and behaviors.
Key results
- LLaRP reaches 42% total success on unseen Language Rearrangement tasks vs 25% for the best baseline (LSTM-Flan) and 22% for ZS-ChatGPT.
- LLaRP achieves 45% on Behavior Generalization vs 28% for LSTM-Flan.
- On Atari, LLaRP's mean, median, and IQM scores are 3x-4x higher than LLaRP-Scratch across 55 games.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaRP instead of the strongest learned baseline, LSTM-Flan | Language Rearrangement evaluation · Unseen tasks (Total) | success rate: 25 { $ \pm$ 1 } -> 42 { $ \pm$ 2 } | Helped |
| Use LLaRP instead of LSTM-Flan | Language Rearrangement evaluation · Behavior Generalization | success rate: 28 { $ \pm$ 1 } -> 45 { $ \pm$ 3 } | Helped |
| Use a pretrained LLaMA backbone rather than training the transformer from scratch (ablation) | LLaRP · Language Rearrangement unseen tasks (Total) | success rate: 17 { $ \pm$ 4 } -> 42 { $ \pm$ 2 } | Helped |
| Scale the LLaMA backbone from 7B to 13B (ablation) | LLaRP · Language Rearrangement evaluation | success rate: +4% | Helped |
| Train with reinforcement learning instead of imitation learning | LLaRP · Language Rearrangement | success rate | Helped |
| Use a LLaRP LLM-based policy rather than LSTM-Flan | Language Rearrangement continual learning · Multiple Rearrangements | environment steps to near-perfect performance: 1.5M vs 500k | Helped |
| Use a pretrained LLM backbone rather than training the transformer from scratch (ablation) | LLaRP · Atari 2600 (55 games) | human-normalized score: between 3x and 4x | Helped |
Surprising findings
- LLaRP-Scratch with the same architecture takes about 10x more environment samples to converge than LLaRP, showing pretrained LLMs help sample efficiency.
- The Multiple Objects and Multiple Rearrangements splits remain near zero for all models, indicating those behaviors remain unsolved.
Limitations
- LLMs are much larger than typical RL policy networks, making training and inference costly.
- Training the action decoder may hinder the policy from exploiting the LLM's world knowledge.
- Multiple-object and multiple-rearrangement generalization remains near zero.
Experimental setup
Datasets: Language Rearrangement (150k train instructions) ReplicaCAD YCB
Benchmarks: Language Rearrangement Atari 2600 (55 games)
Metrics: success rate human-normalized score interquartile mean sample efficiency
Baselines: ZS-ChatGPT ZS-LLaMA ZS-Flamingo LLaRP-Scratch LSTM-Flan LSTM-LLaMA
Builds on: PPO DD-PPO Habitat 2.0 VC-1
Closest papers by meaning
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.89)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.89)
- TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models (ICLR 2026 · similarity 0.89)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.88)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.