Research map / Multi-agent systems and games / Human-AI and goal-conditioned
ADAM: An Embodied Causal Agent in Open-World Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ADAM, a Minecraft embodied agent that learns a causal graph (technology tree) from scratch via LLM-based causal assumptions refined by intervention-based causal discovery, then plans and acts with that graph and MLLM perception.
Problem
Open-world embodied agents in Minecraft struggle to learn structured causal knowledge continuously; existing LLM-based agents rely on prior knowledge and omniscient metadata, and RL agents are black-box and sample-inefficient, hurting interpretability and generalization.
Approach
Combine LLM-based causal discovery (reasoning over interaction records to propose cause/effect items) with embodied intervention-based causal discovery (resampling with cause items removed) to build an ever-growing causal graph from scratch, which a planner/actor/memory controller then uses for task execution, with MLLM-based visual perception instead of metadata.
Key results
- ADAM learns a nearly perfect causal graph with SHD 2±2, versus 10±4 for CDHRL and 24±9 for Reflexion.
- ADAM achieves a 2.2× speedup over VOYAGER-Guided on obtaining diamonds (68±21 vs 75±20 steps for ADAM vs VOYAGER-Guided; 113±41 for VOYAGER per the table), with ADAM Parallel reaching 34±7 steps.
- In a modified-recipe Minecraft, ADAM obtains diamonds (2/3 success for ADAM Parallel-level results) while baselines only reach raw iron, with a 4.6× speedup on raw iron.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use embodied interventions to refine causal assumptions (ablation) | ADAM causal discovery · Minecraft causal graph | SHD: 19 ± 6 -> 2 ± 2 | Helped |
| Add parallel sampling (ablation) | ADAM · Minecraft original, Diamond task | Exploration steps: 68 ± 21 (3/3) -> 34 ± 7 (3/3) | Helped |
| Add multimodal LLM perception (ablation) | ADAM · Environmental-factor tasks | Exploration steps / success: N/A (0/3) -> 21 ± 16 (2/3); N/A (0/3) -> 40 ± 23 (2/3); N/A (0/3) -> 95 (1/3) | Helped |
| Remove temporal modeling and subgraph decomposition (ablation) | ADAM causal discovery · Minecraft causal graph | SHD: 2 ± 2 -> 19 ± 6 | Hurt |
| Use intervention-based causal discovery rather than exhaustive trials (ablation) | ADAM causal discovery · Collecting-action causal subgraphs | Steps and success rate: up to 4.4× speedup; higher success rate | Helped |
| Use GPT-4-turbo-preview for LLM-based causal discovery in the modified environment (ablation) | ADAM causal discovery · Modified Minecraft causal assumptions | R&E (success/all): 0.1 (35/35) | Helped |
| Replace GPT-4-turbo-preview with fine-tuned Llama-2-13B (ablation) | ADAM causal discovery · Minecraft causal assumptions | R&E (success/all): 0.0 (35/35) -> 1.8 (5/35) | Hurt |
| Remove access to metadata (ablation) | VOYAGER · Environmental-factor tasks | Task success: Find a river: 16 ± 8 (2/3) -> N/A (0/3); Gather log near river: 36 (1/3) -> N/A (0/3) | Hurt |
| Learn dependencies from the modified game rather than rely on prior recipes | ADAM versus baseline agents · Modified Minecraft — Diamond | Task success | Helped |
| Use ADAM rather than the strongest baseline | ADAM · Minecraft original, Diamond task | Exploration speed: 2.2× speedup | Helped |
| Use ADAM rather than baseline agents | ADAM · Modified Minecraft, raw_iron task | Exploration speed: 4.6× speedup | Helped |
Surprising findings
- Intervention-based CD yields up to 4.4× acceleration over the non-intervention ablation.
- Strong-reasoning GPT-4-turbo in a modified environment with no prior performs well, while Llama-2-13B fine-tuned on Minecraft QA performs poorly, suggesting reasoning matters more than prior knowledge.
Limitations
- Relies on a discretized action space of 41 actions and 6 movements, requiring manual design of actions and items as graph nodes.
- Collecting and smelting actions have noisy sampling data or complex subgraphs leading to redundant or omitted edges in LLM-based CD.
- Evaluated mainly in a single Minecraft environment with limited task set and only 3 trials per task.
Experimental setup
Datasets: MC-QA (754 Minecraft crafting-recipe QA pairs, constructed by authors) Minecraft interaction records collected by the agent
Benchmarks: Minecraft technology tree (causal graph SHD) Minecraft diamond/iron/stone/wooden tool tasks (exploration steps) environment-factor tasks (find river, gather log near river, smelting iron near grass) modified-recipe Minecraft MC-QA
Metrics: Structural Hamming Distance (SHD) exploration steps success rate redundant/error items (R&E)
Baselines: ReAct Reflexion AutoGPT VOYAGER CDHRL DreamerV3 DEPS
Builds on: VOYAGER CDHRL ReAct Reflexion chain-of-thought prompting
Closest papers by meaning
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.89)
- GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR 2024 · similarity 0.88)
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (CVPR 2024 · similarity 0.88)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.88)
- MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory (ICLR 2025 · similarity 0.88)
- Experience-based Knowledge Correction for Robust Planning in Minecraft (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.