Research map / Detection and segmentation / Open-set and few-shot detection
Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGM, a modular ObjectNav method that pretrains a masked-autoencoder-style ViT to reconstruct masked patches of cropped global maps using episodic observations plus LLM-generated general knowledge, then uses it to imagine unobserved regions of the local semantic map.
Problem
Object goal navigation requires inferring likely target location in unseen environments from partial observations; end-to-end RL lacks geometry, while modular methods lack contextual object reasoning.
Approach
Self-supervised masked map reconstruction (MAE-style) on cropped global maps, fused with LLM-predicted neighboring object categories via a LXMERT cross-modality encoder, generates unobserved regions of the local semantic map so the agent can set a predicted target location as a long-term goal.
Key results
- On Gibson val, SGM achieves 78.0% SR and 44.0% SPL, outperforming 3D-aware (74.5% SR, 42.1% SPL).
- On MP3D val, SGM reaches 37.7% SR and 14.7% SPL, comparable to 3D-aware (34.0% SR, 14.6% SPL).
- On HM3D val, SGM achieves 60.2% SR and 30.8% SPL, above PEANUT re-implementation (59.1% SR, 30.3% SPL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LLM general knowledge to episodic observations (ablation) | SGM · Gibson val | SR: 74.4 -> 78.0 | Helped |
| Add LLM general knowledge to episodic observations (ablation) | SGM · MP3D val | SR: 34.7 -> 37.6 | Helped |
| Add episodic observations alongside general knowledge (ablation) | SGM · Gibson val | SR: 65.1 -> 78.0 | Helped |
| Add episodic observations alongside general knowledge (ablation) | SGM · MP3D val | SR: 25.9 -> 37.6 | Helped |
| Add the early exploration strategy (ablation) | SGM · Gibson val | SR: 74.1 -> 78.0 | Helped |
| Add the early exploration strategy (ablation) | SGM · MP3D val | SR: 33.0 -> 37.6 | Helped |
| Use random rather than connected masking (ablation) | ViT-B map completion · Gibson val | IoU (%): 22.31 -> 25.45 | Helped |
| Use random rather than connected masking (ablation) | ViT-B map completion · Gibson val | SR (%): 73.1 -> 74.1 | Helped |
| Add general knowledge from GPT-4 to ViT-B with random masking (ablation) | SGM map generation · Gibson val | Recall (%): 79.23 -> 87.33 | Helped |
| Increase the visual backbone from ViT-B to ViT-L (ablation) | SGM with random masking · Gibson val | SR (%): 74.1 -> 74.2 | No clear effect |
| Replace GPT-4 with ChatGLM (ablation) | ViT-B SGM with random masking · Gibson val | SR (%): 77.4 -> 78.0 | Helped |
| Use SGM rather than 3D-aware | ObjectNav · Gibson val | SR (%): 74.5 -> 78.0 | Helped |
Surprising findings
- Random masking outperforms connected masking used by prior map completion methods (L2M, SSCNav), contrary to the expectation that local-map completion is the better proxy.
- The choice of LLM (GPT-4 vs ChatGLM) has minimal impact on navigation performance.
Limitations
- Relies on pretrained semantic segmentation models (Mask-RCNN, RedNet) and performance is comparable rather than clearly superior on MP3D.
- Generated map has absolute position deviations for unseen objects.
- Limited training environment diversity for self-supervised map pretraining.
Experimental setup
Datasets: Gibson Matterport3D Habitat-Matterport3D
Benchmarks: Gibson MP3D HM3D
Metrics: SR SPL DTS IoU Recall
Baselines: DD-PPO Red-Rabbit THDA SSCNav EmbCLIP Habitat-Web ANS SemExp PONI L2M Stubborn 3D-aware
Builds on: MAE PONI SemExp LXMERT Fast Marching Method
Closest papers by meaning
- Learning to Map for Active Semantic Goal Navigation (ICLR 2022 · similarity 0.90)
- Distilling LLM Prior to Flow Model for Generalizable Agent’s Imagination in Object Goal Navigation (NeurIPS 2025 · similarity 0.89)
- Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction (ICLR 2024 · similarity 0.87)
- BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · similarity 0.86)
- NaviFormer: A Spatio-Temporal Context-Aware Transformer for Object Navigation (AAAI 2025 · similarity 0.86)
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.