Research map / Reinforcement learning / Visual and representation RL
True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TWOSOME, an online RL framework that turns a frozen LLaMA-7B into a valid-action policy via joint action probabilities with token/word normalization, trained with PPO using shared-frozen-LLM LoRA actor and MLP critic.
Problem
LLMs often fail at simple embodied decision-making because their knowledge is misaligned with environments (invalid actions, wrong dynamics), while RL agents learn from scratch with poor sample efficiency.
Approach
Score each valid action prompt by its LLM joint log-probability (softmax over valid actions) so policies are always valid, correct length bias with token or word normalization, and fine-tune a shared frozen LLM with LoRA actor and MLP critic under PPO, with no pre-collected data.
Key results
- Word-normalized TWOSOME reaches 100% success on all four Overcooked and VirtualHome tasks, while PPO gets 33% on Entertainment and gets stuck in suboptimal policies in Overcooked.
- On eight unseen tasks, TWOSOME achieves 1.00 success on Cheese, Hamburger, Apple Pie, Pizza, Washing Plate and Laundry, versus SayCan at 0.35 to 0.85; Entertainment crossover reaches 0.97 vs 0.04.
- Zero-shot NLP benchmarks after online PPO finetuning remain near the LLaMA-7B baseline (e.g., ARC_C 0.43 vs 0.42, HellaSwag 0.58 vs 0.57, MMLU 0.30-0.32 vs 0.33).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add word normalization to action prompt probabilities (ablation) | TWOSOME with PPO · Tomato Salad | success rate: $0.67 \pm 0.47$ -> $\mathbf{1 \pm 0}$ | No clear effect |
| add word normalization to action prompt probabilities (ablation) | TWOSOME with PPO · Tomato-lettuce Salad | success rate: $1 \pm 0$ -> $\mathbf{1 \pm 0}$ | No clear effect |
| add word normalization to action prompt probabilities (ablation) | TWOSOME with PPO · Food Preparation | success rate: $1 \pm 0$ -> $\mathbf{1 \pm 0}$ | No clear effect |
| add word normalization to action prompt probabilities (ablation) | TWOSOME with PPO · Entertainment | success rate: $0.67 \pm 0.47$ -> $\mathbf{1 \pm 0}$ | Helped |
| add online PPO finetuning (ablation) | TWOSOME with word normalization · Entertainment | success rate: $0.45 \pm 0.05$ -> $\mathbf{1 \pm 0}$ | Helped |
| replace no normalization with word normalization (ablation) | TWOSOME · Overcooked and VirtualHome tasks | training stability | Helped |
| replace word normalization with token normalization (ablation) | TWOSOME · Overcooked tasks | sample efficiency | Hurt |
| use word-normalized TWOSOME instead of SayCan | LLaMA-7B embodied agent · Entertainment (unseen) | success rate: 0.04 -> 0.97 | Helped |
Surprising findings
- Vanilla PPO with a large VirtualHome action space learns nothing, requiring action masking to make progress.
- Without normalization, the joint-probability policy is biased toward shorter action prompts, causing sudden training collapse in most tasks.
Limitations
- Each sampling step requires scoring all valid actions with the LLM, multiplying compute and reducing batch size versus PPO from scratch.
- Experiments are limited to two environments (Overcooked, VirtualHome) with a single LLaMA-7B backbone.
- Reference-model KL regularization was not explored in this work.
Experimental setup
Benchmarks: Overcooked VirtualHome ARC_C HellaSwag PIQA MMLU
Metrics: total return success rate sample efficiency zero-shot accuracy
Baselines: PPO SayCan GLAM Prompt tuning
Builds on: PPO LoRA SayCan CleanRL
Closest papers by meaning
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.88)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.88)
- Controlling Large Language Model with Latent Action (ICML 2025 · similarity 0.88)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.88)
- Embodied CoT Distillation From LLM To Off-the-shelf Agents (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.