Research map / Multi-agent systems and games / LLM agents
JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, Yitao Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal memory with reasoning-based retrieval | JARVIS-1 · Minecraft Diamond group | success rate: 2.42 -> 8.99 | Improved |
| extend game time from 20 to 60 minutes | JARVIS-1 · ObtainDiamondPickaxe | success rate: 6.2% to 12.5% | Improved |
| extend game time from 20 to 60 minutes | VPT RL · ObtainDiamondPickaxe | success rate: 2.5% to 3% | No clear effect |
| add reasoning before retrieval (Text Memory+Reasoning vs Text Memory) (ablation) | JARVIS-1 · Minecraft task subset | success rate | Improved |
| use multimodal state for retrieval instead of text embedding only (ablation) | JARVIS-1 · Minecraft task subset | success rate | Improved |
| add memory module to the agent (ablation) | JARVIS-1 · Minecraft task subset | success rate | Improved |
| replace GPT-4 with ChatGPT as planner LM (ablation) | JARVIS-1 · Minecraft task subset | success rate | No clear effect |
| finetune LLaMA2-13B on Minecraft text vs pretrained LLaMA2-70B (ablation) | JARVIS-1 · Minecraft task subset | success rate | Improved |
| use GPT-generated self-improve curriculum instead of human-written or random (ablation) | JARVIS-1 · ObtainDiamondPickaxe (60 min) | success rate | Improved |
| add interactive re-planning with feedback (Inner Monologue vs GPT) | GPT · Minecraft Wood group | success rate: 27.30 -> 60.15 | Improved |
Later papers that compared with this method
2 later accepted papers report JARVIS-1 as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Experience Transfer for Multimodal LLM Agents in Minecraft Game (CVPR 2026 · better on 13 of 13 shared results)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · better on 8 of 8 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.90)
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (CVPR 2024 · similarity 0.90)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.90)
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.89)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.88)
- MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
