Research map / Vision-language and multimodal / Human and embodied modeling
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EmbodiedScan, a large multi-modal ego-centric 3D indoor perception dataset with oriented boxes, occupancy and language prompts, plus the Embodied Perceptron baseline and benchmarks.
Problem
Existing 3D scene understanding research mostly uses global scene-level inputs (point clouds/meshes) and datasets with ego-centric RGB-D are too small or lack comprehensive annotations, which limits holistic perception for embodied agents.
Approach
A large ego-centric RGB-D dataset re-purposed from ScanNet, 3RScan and Matterport3D with SAM-assisted oriented 3D box annotations, dense semantic occupancy and SR3D-style language prompts, paired with a baseline that fuses any number of RGB-D views via isomorphic multi-level multi-modal fusion.
Key results
- Embodied Perceptron (RGB-D) reaches 16.85 AP25 overall on the large-vocabulary multi-view 3D detection benchmark, outperforming FCAF3D+painting (15.10) and the depth-only variant.
- Multi-modality baseline achieves 20.79 mIoU on continuous occupancy prediction versus 14.44 for depth-only and 10.43 for camera-only.
- Multi-view 3D visual grounding: Ours reaches 25.72 overall AP25, above L3Det (23.07) and BUTD-DETR (22.14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RGB-D rather than depth-only input | Embodied Perceptron · Continuous 3D detection, large-vocabulary | AP25: 17.16 -> 19.07 | Helped |
| use RGB-D rather than depth-only input | Embodied Perceptron · Continuous semantic occupancy prediction | mIoU: 14.44 -> 20.79 | Helped |
| use RGB-D rather than depth-only input | Embodied Perceptron · Multi-view semantic occupancy prediction | mIoU: 15.56 -> 19.97 | Helped |
| use the proposed model instead of L3Det | Multi-view 3D visual grounding · EmbodiedScan | AP25: 23.07 -> 25.72 | Helped |
| add oriented-box prediction rather than use axis-aligned boxes (ablation) | FCAF3D · ScanNet 18-class detection | AP25: 70.17 -> 61.87 | Hurt |
| add multi-view input to oriented-box detection (ablation) | FCAF3D · ScanNet 18-class detection | AP25: 61.87 -> 59.95 | Hurt |
| add multi-view input to oriented-box detection (ablation) | FCAF3D · ScanNet 18-class detection | AP50: 47.30 -> 43.33 | Hurt |
| evaluate on real images instead of rendered images (ablation) | Embodied Perceptron trained on rendered images · Multi-view 3D detection, large-vocabulary validation | Overall AP: 22.11 -> 18.72 | Hurt |
| train on real rather than rendered images (ablation) | Embodied Perceptron · Multi-view 3D detection, large-vocabulary validation | Overall AP: 18.72 -> 21.98 | Helped |
| train on EmbodiedScan rather than ScanNet (ablation) | Embodied Perceptron · ScanNet 3D detection validation | Overall AP: 20.28 -> 23.02 | Helped |
| train on EmbodiedScan rather than ScanNet (ablation) | Embodied Perceptron · EmbodiedScan 3D detection validation | Overall AP: 10.92 -> 16.85 | Helped |
Surprising findings
- Models trained on rendered images lose performance when transferred to real images (5.99% AP drop on head categories), and rendered-only training underperforms real-only training.
- Switching ScanNet detection from axis-aligned to oriented boxes drops AP25 from 70.17 to 61.87, showing orientation estimation is challenging.
Limitations
- Large-vocabulary tail categories perform poorly, suggesting limited data per category
- Visual grounding performance remains much lower than prior benchmarks due to changed input format and annotations
- ARKitScenes not yet included in the dataset
Experimental setup
Datasets: ScanNet 3RScan Matterport3D SR3D SUN RGB-D LVIS
Benchmarks: EmbodiedScan multi-view 3D detection EmbodiedScan continuous 3D detection EmbodiedScan occupancy prediction EmbodiedScan monocular 3D detection EmbodiedScan multi-view 3D visual grounding ScanNet
Metrics: AP25 AP50 AR25 AR50 mIoU
Baselines: ImVoxelNet VoteNet FCAF3D ImVoteNet FCOS3D OccNet SurroundOcc ScanRefer BUTD-DETR L3Det
Builds on: FCAF3D SR3D SAM ImVoxelNet BERT
Closest papers by meaning
- SPA: 3D Spatial-Awareness Enables Effective Embodied Representation (ICLR 2025 · similarity 0.91)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence (CVPR 2025 · similarity 0.90)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.89)
- Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.