Research map / Multi-agent systems and games / LLM agents
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes an MLLM (LLaVA-7B) as a Bradley-Terry trajectory scorer using interaction-derived preference pairs, plus GPT-4o-based trajectory abstraction, to retrieve effective expert trajectories for embodied agents in AI2-THOR and LEGENT.
Problem
Retrieval of multimodal trajectories for MLLM embodied agents relies on surface-level text/image similarity, which does not reflect how effective a trajectory is for the specific task at hand.
Approach
Use the agent's own success rates when conditioned on different reference trajectories to build pairwise preference data, train an MLLM retriever with a Bradley-Terry score head to rank trajectories by effectiveness, and compress trajectories into milestone abstractions generated by an MLLM.
Key results
- On AI2-THOR, MART reaches 0.40 SR vs 0.26 for the best baseline (LLaVA-Plain) and 0.18 for Plain-Agent.
- On LEGENT, MART reaches 0.87 SR vs 0.75 for Similarity+LLaVA and RAP.
- Ablation: removing Trajectory Abstraction drops AI2-THOR SR from 0.40 to 0.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use interactive preference-trained multimodal retrieval instead of LLaVA-Plain | MART · AI2-THOR | SR: 0.26 -> 0.40 | Helped |
| use interactive preference-trained multimodal retrieval instead of LLaVA-Plain | MART · AI2-THOR | AS: 144.18 -> 123.19 | Helped |
| use interactive preference-trained multimodal retrieval instead of similarity-based retrieval | MART · LEGENT | SR: 0.75 -> 0.87 | Helped |
| use interactive preference-trained multimodal retrieval instead of RAP | MART · ReALFRED | SR: 0.27 -> 0.37 | Helped |
| add Trajectory Abstraction (ablation) | MART · AI2-THOR | SR: 0.31 -> 0.40 | Helped |
| add Trajectory Abstraction (ablation) | MART · LEGENT | SR: 0.77 -> 0.87 | Helped |
| use unified MART retrieval instead of similarity-based retrieval plus MART ranking (ablation) | MART · AI2-THOR | SR: 0.34 -> 0.40 | Helped |
| use unified MART retrieval instead of similarity-based retrieval plus MART ranking (ablation) | MART · LEGENT | SR: 0.77 -> 0.87 | Helped |
| add Trajectory Abstraction (ablation) | MART · AI2-THOR | AS: 130.26 -> 123.19 | Helped |
| add Trajectory Abstraction (ablation) | MART · LEGENT | AS: 18.48 -> 13.81 | Helped |
Surprising findings
- Plain embedding similarity correlates weakly with success rate, and similarity-based retrieval often pulls in misleading trajectories.
- Similarity+LLaVA (SL) and RAP sometimes underperform plain agent in AI2-THOR (RAP 0.22 vs PA 0.18 SR is higher, but SL 0.24 is also above PA; on the Come-2room LEGENT type RAP drops to 0.46).
Limitations
- MLLM context window restricts retrieval to one-shot, single-trajectory prompts.
- Small memory sizes (tens of trajectories) and results limited to household domain.
- Lack of detailed ablations on retriever base models and other components, as acknowledged by authors.
Experimental setup
Datasets: AI2-THOR task set (45 training tasks, 28 test tasks) LEGENT task set (40 training, 32 test) ReALFRED pick_and_place (30 train, 20 test) multi-image captioning data generated by LLaVA
Benchmarks: AI2-THOR (custom ALFRed/ALFWorld-based setting) LEGENT ReALFRED
Metrics: success rate sub-task success rate average steps
Baselines: Plain-Agent (GPT-4o) LLaVA-Plain Similarity+LLaVA RAP Similarity+FTM
Builds on: LLaVA Bradley-Terry reward modeling RAP ALFRed ALFWorld
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.87)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.87)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.87)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.87)
- Retrieval-Augmented Embodied Agents (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.