Research map / 3D vision / 3D generation
3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-Mem, a 3D scene memory of multi-view memory snapshots plus frontier snapshots that a VLM agent uses for embodied exploration, question answering, and lifelong navigation.
Problem
Object-centric 3D scene graphs oversimplify spatial relationships and cannot represent unexplored regions, which limits embodied exploration, reasoning, and lifelong memory in complex 3D environments.
Approach
Represent explored regions as a compact set of multi-view snapshot images that each capture a cluster of co-visible objects plus background context, and represent unexplored regions as frontier snapshots, so a VLM can reason over and explore using raw images, with incremental construction and prefiltering for retrieval.
Key results
- On A-EQA (184-question subset), 3D-Mem reaches 52.6 LLM-Match and 42.0 SPL, versus 46.9 / 23.4 for Explore-EQA.
- On EM-EQA, 3D-Mem reaches 57.2 LLM-Match using about 3.1 frames, outperforming Multi-Frame (48.1 with 3.0 frames).
- On GOAT-Bench Val Unseen subset, 3D-Mem reaches 69.1% success rate and 48.9 SPL, above ConceptGraph w/ frontier snapshots (61.5 / 45.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3D-Mem snapshots instead of ConceptGraph object crops | 3D-Mem with frontier snapshots · A-EQA (184-question subset) | LLM-Match: 47.2 -> 52.6 | Helped |
| use 3D-Mem snapshots instead of ConceptGraph object crops | 3D-Mem with frontier snapshots · A-EQA (184-question subset) | LLM-Match SPL: 33.3 -> 42.0 | Helped |
| use 3D-Mem snapshots instead of linearly sampled frames | 3D-Mem · EM-EQA | LLM-Match: 48.1 -> 57.2 | Helped |
| retain scene memory across navigation subtasks instead of clearing it after each subtask (ablation) | 3D-Mem with GPT-4o · GOAT-Bench Val Unseen subset | Success Rate: 58.6 -> 69.1 | Helped |
| retain scene memory across navigation subtasks instead of clearing it after each subtask (ablation) | 3D-Mem with GPT-4o · GOAT-Bench Val Unseen subset | SPL: 38.5 -> 48.9 | Helped |
| retain scene memory across navigation subtasks instead of clearing it after each subtask (ablation) | 3D-Mem with an open-sourced VLM · GOAT-Bench Val Unseen subset | Success Rate: 40.6 -> 49.6 | Helped |
| retain scene memory across navigation subtasks instead of clearing it after each subtask (ablation) | 3D-Mem with an open-sourced VLM · GOAT-Bench Val Unseen subset | SPL: 14.6 -> 29.4 | Helped |
Limitations
- Evaluated on subsets (184 of 557 A-EQA questions; 1/10 of GOAT-Bench Val Unseen) due to resource limits
- Relies on closed VLM API (GPT-4o) for reasoning
- Navigation assumes a collision-free planner in Habitat-sim
Experimental setup
Datasets: A-EQA (OpenEQA) EM-EQA (OpenEQA) GOAT-Bench
Benchmarks: A-EQA EM-EQA GOAT-Bench
Metrics: LLM-Match LLM-Match SPL Success Rate SPL frame efficiency
Baselines: ConceptGraphs w/ frontier snapshots Explore-EQA Multi-Frame LLaVA-1.5 frame captions GPT-4o blind 3D-Mem w/o memory Modular GOAT SenseAct-NN
Builds on: ConceptGraphs ExploreEQA frontier-based exploration OpenEQA
Closest papers by meaning
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.89)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
- Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization (ICLR 2026 · similarity 0.88)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.88)
- 3D-SPATIAL MULTIMODAL MEMORY (ICLR 2025 · similarity 0.88)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.