Research map / Reinforcement learning / Visual and representation RL
LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LARM, a lightweight (<5B) auto-regressive policy that outputs skill tokens and is trained with referee RL, where GPT-4 provides auxiliary per-step rewards to counter reward vanishing in long-horizon Minecraft tasks.
Problem
Long-horizon embodied agents trained with classic RL receive vanishing feedback because rewards arrive only after long action chains; giant LLM agents are too costly to deploy on embodied hardware.
Approach
Build the policy on a lightweight LLM (TinyLLaVA-3.1B with LoRA) that directly outputs a skill token, and train it with PPO plus an auxiliary reward from a giant LLM referee (GPT-4/GPT-4o) that judges each action's immediate contribution to the target task.
Key results
- On MineDojo, LARM reaches 0.93 on Harvest stick and 0.97 on Harvest bowl, outperforming RL-GPT (0.65) and other baselines on tasks where they report results.
- On Mineflayer, LARM completes the enchanted diamond sword in 16/30 trials, the first reported success on this achievement where Voyager, STEVE, and AutoGPT get 0/3.
- Ablation: environment reward only gets 0.00 success on iron sword; adding GPT-4o referee reward with four outcome categories (ER+AR4) raises it to 0.27 and stone sword from 0.10 to 0.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 auxiliary rewards instead of environment reward alone (ablation) | Referee RL · MineDojo, Harvest stick | success rate: 0.20 -> 0.93 | Helped |
| Use GPT-4 auxiliary rewards instead of environment reward alone (ablation) | Referee RL · MineDojo, Harvest wooden sword | success rate: 0.13 -> 0.70 | Helped |
| Use GPT-4 auxiliary rewards instead of environment reward alone (ablation) | Referee RL · MineDojo, Harvest stone sword | success rate: 0.10 -> 0.40 | Helped |
| Use GPT-4 auxiliary rewards instead of environment reward alone (ablation) | Referee RL · MineDojo, Harvest iron sword | success rate: 0.00 -> 0.27 | Helped |
| Replace GPT-4 auxiliary feedback with LLaVA-7B feedback (ablation) | Referee RL · MineDojo, Harvest stick | success rate: 0.93 -> 0.30 | Hurt |
| Use four-category feedback instead of judging only whether the outcome is positive (ablation) | Referee RL · MineDojo, Harvest stick | success rate: 0.80 -> 0.93 | Helped |
| Use four-category feedback instead of judging only whether the outcome is positive (ablation) | Referee RL · MineDojo, Harvest iron sword | success rate: 0.07 -> 0.27 | Helped |
| Increase the LLM base from TinyLLaVA-0.5B to TinyLLaVA-3.1B (ablation) | LARM · MineDojo, Harvest stick | success rate: 0.80 -> 0.83 | Helped |
| Add webpage-data pretraining (ablation) | LARM with TinyLLaVA-3.1B · MineDojo, Harvest stick | success rate: 0.83 -> 0.93 | Helped |
| Add webpage-data pretraining (ablation) | LARM with TinyLLaVA-3.1B · MineDojo, Harvest iron sword | success rate: 0.13 -> 0.27 | Helped |
Surprising findings
- A lightweight LLM (LLaVA-7B) used as the referee produced poor auxiliary rewards, giving worse results than no auxiliary reward on some tasks.
- Binary-style referee reward (only positive/negative outcome) performed worse than the four-category referee reward, since correct actions with no immediate positive outcome were penalized.
Limitations
- Referee reward relies on giant closed LLMs (GPT-4/GPT-4o), so cost and reproducibility depend on API access.
- Skills are predefined (RL-based skills in MineDojo or Mineflayer APIs); skill generation is not studied.
Experimental setup
Datasets: MineDojo tasks Mineflayer environment 34G Minecraft webpage data crawled from Wiki
Benchmarks: MineDojo harvest tasks Mineflayer achievements (wooden to enchanted diamond sword)
Metrics: success rate
Baselines: MineAgent Plan4MC LLaMA-Rider RL-GPT AutoGPT Voyager STEVE
Builds on: PPO Generalized Advantage Estimation TinyLLaVA CLIP STEVE
Closest papers by meaning
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.87)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.87)
- WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.87)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.86)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.86)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.