Research map / Vision-language and multimodal / Robot action and navigation
An Embodied Generalist Agent in 3D World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LEO, an embodied generalist agent that takes egocentric 2D images, object-centric 3D point clouds and text to perform 3D captioning, QA, dialogue, planning, navigation and manipulation via a unified LLM (Vicuna-7B + LoRA) trained in two stages with LLM-generated 3D-language data.
Problem
Generalist models mostly operate on 2D images and rarely handle 3D-defined tasks such as 3D grounding, embodied reasoning and acting; 3D-language data is scarce.
Approach
A unified sequence-prediction interface that tokenizes egocentric 2D image tokens, object-centric 3D tokens (Mask3D proposals + Spatial Transformer) and text/discretized action tokens into a decoder-only LLM, trained in 3D-VL alignment then 3D VLA instruction tuning, with scene-graph-based, object-centric CoT LLM data generation plus regex/scene-graph refinement.
Key results
- On Scan2Cap (val) LEO reaches 72.4 CIDEr vs 66.9 for 3D-VisTA; on ScanQA (val) it reaches 101.4 CIDEr and 24.5 EM@1 (47.6 with refined EM) vs 69.6 CIDEr for 3D-VisTA.
- On CLIPort manipulation LEO matches the best seen-task score on packing-google-objects-seq (98.8 vs CLIPort single 98.0 on put-blocks seen) and reaches 75.2 on unseen packing tasks, tying CLIPort single.
- On object navigation LEO achieves 15.2 SPL on MP3D-val vs 2.2 for Habitat-web (shortest) and 19.1 zero-shot SPL on HM3D vs ZSON's 12.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the 3D vision-language alignment stage (ablation) | LEO · Scan2Cap (ScanNet) | sentence similarity: 62.8 -> 65.4 | Helped |
| Train on broad scenes and tasks instead of ScanNet scenes only (ablation) | LEO · 3RDialog | sentence similarity: 25.5 -> 73.3 | Helped |
| Include embodied acting tasks in instruction tuning (ablation) | LEO · Scan2Cap (ScanNet) | sentence similarity: 65.4 -> 65.3 | Hurt |
| Add dialogue and planning data (ablation) | LEO · Human preference: Answerable | TrueSkill: 24.4±1.3 -> 25.6±1.3 | Helped |
| Add dialogue and planning data (ablation) | LEO · Human preference: Unanswerable | TrueSkill: 23.1±1.4 -> 26.8±1.4 | Helped |
| Add dialogue and planning data (ablation) | LEO · Human preference: NLP | TrueSkill: 23.4±1.4 -> 26.6±1.4 | Helped |
| Augment training with negative object-existence examples (ablation) | LEO · 3RScan object-existence questions | overall EM: 0.34 -> 0.85 | Helped |
| Augment training with negative object-existence examples (ablation) | LEO · ScanNet (0-shot) object-existence questions | overall EM: 0.43 -> 0.83 | Helped |
| Use object-centric chain-of-thought prompting (ablation) | LLM-assisted 3D data generation · Object counting questions | answer accuracy: 0.5741 -> 0.7802 | Helped |
| Add refinement procedures to generated data (ablation) | LLM-assisted 3D data generation · Counting questions | answer accuracy: 78.0 -> 100.0 | Helped |
Surprising findings
- Adding embodied acting (VLA) tasks to training degrades 3D VL performance (e.g., Scan2Cap 65.4 w/o Act vs 65.3 with, and 3RQA 51.9 vs 51.3 with lower overall trends).
- Imbalanced existence QA data induces a strong 'Yes' bias; augmenting with negatives raised overall object-existence accuracy from 0.34 to 0.85 on 3RScan.
Limitations
- Generalization to novel scenes is limited; zero-shot results on 3RScan-trained scenes are weaker.
- Notable gap between VL learning and embodied action control; joint training can hurt VL performance.
- Navigation uses only truncated past actions, lower success rate than recurrent baselines.
Experimental setup
Datasets: Objaverse Cap3D ScanNet 3RScan ScanScribe ReferIt3D 3DSSG Scan2Cap ScanQA SQA3D MP3D CLIPort
Benchmarks: Scan2Cap ScanQA SQA3D CLIPort manipulation MP3D ObjNav HM3D ObjNav
Metrics: CIDEr BLEU-4 METEOR ROUGE sentence similarity exact match success rate SPL TrueSkill
Baselines: Scan2Cap 3DJCG Vote2Cap-DETR ScanQA 3D-VisTA 3D-LLM CLIPort Habitat-web ZSON
Builds on: LLaVA BLIP-2 3D-VisTA Chat-3D
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.90)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.90)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.90)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.89)
- Towards Learning a Generalist Model for Embodied Navigation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.