Research map / Vision-language and multimodal / Robot action and navigation
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiPLY, an embodied multisensory LLM that actively interacts with 3D scenes via action and state tokens, trained on the 500k-sample Multisensory Universe dataset.
Problem
Multimodal LLMs passively absorb sensory inputs and cannot actively interact with objects in 3D environments to collect visual, audio, tactile, and thermal information.
Approach
Encode the 3D scene as abstracted object-centric representations, add action tokens (select, navigate, observe, touch, hit) that let the LLM act in a simulator, and feed the resulting multisensory observations back via state tokens for step-by-step generation.
Key results
- Object retrieval accuracy of 56.7 for MultiPLY vs. 48.9 for the best baseline PointBind-LLM (Finetuned).
- Tool use accuracy of 41.6 for MultiPLY vs. 32.1 for PointBind-LLM (Finetuned).
- Task decomposition success rate of 30.2 for MultiPLY vs. 22.4 for 3D-LLM with interaction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MultiPLY instead of PointBind-LLM (Finetuned) | Object retrieval · Object retrieval | Retrieval Accuracy: 48.9 -> 56.7 | Helped |
| Use MultiPLY instead of MultiPLY-2D | Tool use · Tool use | Accuracy: 36.3 -> 41.6 | Helped |
| Use MultiPLY instead of PointBind-LLM (Finetuned) | Multisensory captioning · Multisensory captioning | BLEU1: 36.7 -> 48.9 | Helped |
| Use MultiPLY instead of PointBind-LLM (Finetuned) | Multisensory captioning · Multisensory captioning | BLEU4: 14.5 -> 20.1 | Helped |
| Use MultiPLY instead of PointBind-LLM (Finetuned) | Multisensory captioning · Multisensory captioning | METEOR: 15.1 -> 24.2 | Helped |
| Use MultiPLY instead of 3D-LLM w Interaction | Task decomposition · Task decomposition | success rate: 22.4 -> 30.2 | Helped |
Surprising findings
- Models without environment interaction perform very poorly on task decomposition (e.g., LLaVA 4.0), largely due to hallucinated objects.
Limitations
- Relies on pre-defined navigation and control policies rather than learned ones.
- Evaluated only in simulation with pre-defined sensors and objects.
- Absolute performance remains modest (e.g., 30.2% task decomposition success).
Experimental setup
Datasets: Multisensory Universe HM3D ObjectFolder Objaverse AudioSet AudioCaps
Benchmarks: Multisensory Universe held-out tasks (object retrieval, tool use, multisensory captioning, task decomposition)
Metrics: retrieval accuracy tool use accuracy BLEU1 BLEU4 METEOR task success rate
Baselines: LLaVA 3D-LLM PointBind-LLM ConceptGraph+CLIP ConceptGraph+CLAP ConceptGraph+ImageBind ConceptGraph+PointBind
Builds on: LLaVA ConceptGraphs 3D-LLM CLAP ImageBind
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.88)
- 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination (CVPR 2025 · similarity 0.88)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.88)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.