Research map / Multi-agent systems and games / LLM agents

Experience Transfer for Multimodal LLM Agents in Minecraft Game

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Echo, a transfer-oriented memory framework for multimodal LLM agents in Minecraft that organizes experience into five transfer axes (structure, attribute, process, function, interaction) and uses in-context analogy learning to reuse it.

Problem

Existing Minecraft agents treat memory as a passive store and transfer knowledge only superficially, so they relearn patterns that recur across tasks and worlds, slowing from-scratch learning.

Approach

Decompose reusable knowledge into five explicit transfer dimensions encoded in a Contextual State Descriptor (CSD) and retrieve top-K analogous past tasks across these axes to build in-context analogy prompts that propose new tasks, with only successful trajectories written back to memory.

Key results

• Echo achieves roughly 1.3x–1.7x speed-up in reaching object-unlocking milestones versus MP5, Voyager, JARVIS-1, and MrSteve in from-scratch Minecraft learning.
• At episode 30 of continuous learning, Echo reaches 45% success versus 43% for MP5, 35% for JARVIS-1, 33% for MrSteve, and 18% for Voyager.
• Echo (8-shot) reaches 62.5/92.5 on Bed and 50.0/80.0 on BridgeEq success at 10/30 episodes, outperforming baselines on most task families; 2-shot is described as competitive rather than uniformly best.

Surprising findings

• Echo starts slower than baselines in early episodes and only overtakes them in the mid-to-late phase, which the authors attribute to cold-start knowledge accumulation.
• Echo is less effective in information-sparse settings where active exploration is needed, since it relies more on prior knowledge and retrieval.

Limitations

• Evaluated mainly in Minecraft, an idealized environment with simple consistent rules; real-world applicability is unvalidated.
• Slower initial learning rate and weaker performance in information-sparse exploration settings.
• The paper does not name the specific MLLM backbone used, limiting reproducibility.

Experimental setup

Datasets: multimodal task instructions historical execution traces verifier feedback

Benchmarks: Minecraft from-scratch crafting tasks (Recipe, Functional Eq., Crafting Chain, Utility Blocks)

Metrics: Success@0→10 Success@0→30 item unlocking count delta success rate

Baselines: Voyager MrSteve MP5 JARVIS-1 DEPS

Builds on: ICAL Voyager JARVIS-1 MrSteve

Closest papers by meaning

• Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.88)
• MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory (ICLR 2025 · similarity 0.88)
• Experience-based Knowledge Correction for Robust Planning in Minecraft (ICLR 2026 · similarity 0.87)
• Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.86)
• OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.86)
• XSkill: Continual Learning from Experience and Skills in Multimodal Agents (ICML 2026 · similarity 0.86)