Research map / Multi-agent systems and games / Human-AI and goal-conditioned
World Model Implanting for Test-time Adaptation of Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WorMI, which implants independently trained domain-specific world models into a frozen LLM reasoning policy via prototype-based retrieval and meta-learned world-wise compound attention, evaluated zero- and few-shot on VirtualHome and ALFWorld.
Problem
Enabling LLM-based embodied agents to adapt zero-shot and few-shot to unseen household domains (tasks and scenes) without extensive data collection or retraining.
Approach
Test-time composition of multiple pre-trained domain world models with a frozen LLM reasoning model: relevant world models are retrieved via Wasserstein distance between k-center object-embedding prototypes, then fused by hierarchical (world-level and reasoning-level) cross-attention that is meta-learned, so models can be implanted or removed without retraining.
Key results
- On VirtualHome unseen tasks and scenes (zero-shot), WorMI reaches 66.12% SR versus 45.71% for SayCanPay, with PS 15.17 versus 19.04.
- On ALFWorld unseen tasks and scenes (zero-shot), WorMI reaches 51.67% SR versus 39.66% for SayCanPay.
- Few-shot VirtualHome average SR is 77.51% for WorMI versus 50.93% for SayCanPay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WorMI instead of SayCanPay | VirtualHome zero-shot adaptation to unseen tasks and scenes · VirtualHome, unseen tasks & unseen scenes | SR: 20.41% increase | Helped |
| Use WorMI instead of SayCanPay | ALFWorld zero-shot adaptation to unseen tasks and scenes · ALFWorld, unseen tasks & unseen scenes | SR: 12.01% increase | Helped |
| Use WorMI instead of SayCanPay | VirtualHome few-shot adaptation · VirtualHome, few-shot average | SR: 26.58% gain | Helped |
| Use WorMI instead of SayCanPay | ALFWorld few-shot adaptation · ALFWorld, few-shot average | SR: 19.16% gain | Helped |
| Replace prototype-based retrieval with using all world models (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 66.12% -> 48.63% | Hurt |
| Replace prototype-based retrieval with random subset selection (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 66.12% -> 62.04% | Hurt |
| Replace compound attention with concatenation of world model representations (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 66.12% -> 47.79% | Hurt |
| Replace compound attention with summation of world model representations (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 66.12% -> 56.47% | Hurt |
| Use only the single most relevant world model without fusion (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 66.12% -> 42.75% | Hurt |
| Increase the number of world models from 1 to 2 (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 42.75% -> 62.94% | Helped |
| Increase the number of world models from 2 to 4 (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 62.94% -> 61.96% | Mixed |
| Increase the number of world models from 4 to 6 (ablation) | WorMI · VirtualHome, unseen tasks & scenes | SR: 61.96% -> 48.23% | Hurt |
Surprising findings
- Adding more world models beyond 3 (up to 6) degrades performance sharply, and with 50% adversarial untrained world models SR falls to 38.03%.
- SayCanPay performance barely changes across LLM sizes, suggesting its affordance and cost models drive most of its performance rather than the LLM.
Limitations
- Inference over multiple domain-specific world models increases computational overhead.
- Performance depends on the strength of the underlying LLM reasoning model.
- Results are limited to two simulated benchmarks with a small number of domains.
Experimental setup
Datasets: VirtualHome episodes collected by authors (1,023) ALFWorld (3,554 episodes, CL-ALFRED settings)
Benchmarks: VirtualHome ALFWorld
Metrics: success rate pending steps goal-conditioned success rate
Baselines: ZSP LLM+FT LLM-Planner SayCanPay
Builds on: CALM SayCanPay LLM-Planner MAML
Closest papers by meaning
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.89)
- TANGO: Training-free Embodied AI Agents for Open-world Tasks (CVPR 2025 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.88)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.87)
- Test-Time Adaptation for LLM Agents via Environment Interaction (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.