Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Retrieval-Augmented Embodied Agents

CVPR 2024 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Embodied agent learning · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Retrieval-Augmented Embodied Agents (RAEA), which retrieve policies from an Open X-Embodiment-based memory bank via a multi-modal ImageBind retriever and feed them into a transformer policy generator trained with behavior cloning.

Problem

Embodied agents rely on large amounts of training data and implicit parameter knowledge, so they scale poorly in low-data manipulation settings; the paper asks how to use external multi-embodiment robot data to improve manipulation precision.

Approach

Combine a multi-modal dense policy retriever (ImageBind-based mixed-modal encoders over instructions and observations) with a policy generator that uses cross-attention over tokenized retrieved policies (observations, instructions, actions, proprioception) to condition behavior cloning, plus diversity-oriented retrieval (duplicate filtering at 0.9 similarity and ~70% query token dropout).

Key results

• On Maniskill-2 with image-only observations, RAEA reached 56.7% on PickCube versus 41.3% for Swin-Base, and 93.6% on StackCube versus 86.3% for ResNet152.
• In real-world experiments, the full RAEA with image+video+text instruction reached 69% success, up from 54% for the text-image baseline configuration and 63% for image+video.
• Removing action and proprioception from retrieved policies dropped real-world success from 54% to 36%; Franka-only memory bank gave 48% versus 54% with all embodiments.

Measured effects reported in the paper

Surprising findings

• Retrieved policies that include action and proprioception tokens matter: omitting them dropped real-world success from 54% to 36%.
• Using only Franka-embodiment data in the memory bank gave a smaller decline (48%) than expected relative to the full multi-embodiment bank (54%), though the authors attribute the gap to environment and command diversity.

Limitations

• Real-world evaluation uses a single Franka robot with a custom dataset of 60 tasks, limiting generality.
• Retrieval diversity heuristics (0.9 threshold, 70% token dropout) are hand-set and only lightly justified.
• Comparisons on Franka Kitchen and Meta-World use a few-shot BC setup, and baselines' hyperparameters and training details are not fully reported.

Experimental setup

Datasets: Open X-Embodiment Franka Kitchen Meta-World Maniskill-2

Benchmarks: Franka Kitchen Meta-World Maniskill-2 real-world Franka tasks

Metrics: success rate

Baselines: R3M BLIP-2 EmbodiedGPT ResNet152 Swin Transformer Base

Builds on: ImageBind Retrieval-augmented generation Open X-Embodiment

Closest papers by meaning

• Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization (ICLR 2026 · similarity 0.89)
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents (ICLR 2025 · similarity 0.87)
• Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following (NeurIPS 2024 · similarity 0.86)
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.86)
• ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.86)