Research map / Multi-agent systems and games / LLM agents

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

NeurIPS 2024 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Optimus-1, a Minecraft agent with a Hybrid Multimodal Memory (a Hierarchical Directed Knowledge Graph plus an Abstracted Multimodal Experience Pool) that guides knowledge-aware planning and experience-driven reflection, evaluated on a 67-task long-horizon benchmark.

Problem

Open-world agents such as LLM/MLLM-based planners struggle to complete long-horizon tasks in Minecraft because they lack structured world knowledge and multimodal experience to guide planning and reflection.

Approach

Store world knowledge as an explicit directed graph (HDKG) retrieved by topological sorting for planning, and store summarized multimodal experiences of both successes and failures (AMEP) for in-context reflection, with the MLLM kept frozen and no parameter updates.

Key results

• Optimus-1 achieves 22.26% overall average success rate versus 16.89% for Jarvis-1, 5.39% for DEPS, and 0% for GPT-4V and GPT-3.5 on the 67-task benchmark (human-level 36.41%).
• Optimus-1 reaches 93.6% higher... specifically 98.60% SR on the Wood group versus 93.76% for Jarvis-1.
• With hybrid multimodal memory, open-source MLLMs (DeepSeek-VL, InternLM-XComposer2-VL) improve 2 to 6 times across task groups, outperforming the GPT-4V baseline.

Measured effects reported in the paper

Surprising findings

• GPT-4V and GPT-3.5 without memory achieve 0% success on Iron and all harder groups, showing raw MLLMs lack Minecraft knowledge.
• Retrieving both successful and failed cases from AMEP outperforms retrieving successful cases alone, suggesting failure cases are useful references.

Limitations

• Weak at complex tasks such as beating the Ender Dragon or building a house, limited by STEVE-1's instruction following and action generation.
• Relies on a modular pipeline with separate planner, reflector and action controller rather than an end-to-end vision-language-action model.

Experimental setup

Datasets: Minecraft free exploration experiences teacher-provided task plans

Benchmarks: 67-task Minecraft long-horizon benchmark (7 groups) Voyager benchmark MP5 benchmark DEPS benchmark

Metrics: success rate average steps average time

Baselines: GPT-3.5 GPT-4V DEPS Jarvis-1 Human-level baseline

Builds on: STEVE-1 MineCLIP Voyager DEPS Jarvis-1

Closest papers by meaning

• Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (ICLR 2026 · similarity 0.90)
• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.89)
• Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.88)
• Experience Transfer for Multimodal LLM Agents in Minecraft Game (CVPR 2026 · similarity 0.88)
• ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.88)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.88)