Research map / Detection and segmentation / Open-set and few-shot detection
SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SG-Nav, a zero-shot object navigation framework that builds an online hierarchical 3D scene graph, prompts an LLM with hierarchical chain-of-thought over subgraphs to score frontiers, and adds graph-based re-perception to reject false-positive goal detections.
Problem
Zero-shot object-goal navigation with LLMs, where prompting with only nearby object category names lacks scene context and the LLM outputs frontier probabilities abstractly, leading to unexplainable and weak decisions; detector false positives also cause irreversible navigation.
Approach
Represent the observed scene as a hierarchical 3D scene graph (objects, groups, rooms) with LLM-friendly prompts, use hierarchical chain-of-thought over object-centered subgraphs to predict goal proximity per frontier, and accumulate credibility of detected goals over multiple views to re-perceive false positives.
Key results
- On MP3D, SG-Nav-GPT reaches 40.2 SR and 16.0 SPL, exceeding the zero-shot VLFM (36.2 SR) and the supervised SemEXP (36.0 SR).
- SG-Nav-GPT achieves 54.0 SR on HM3D and 47.5 SR on RoboTHOR, improving over the best prior zero-shot entries by more than 10 points SR as claimed.
- Removing the re-perception mechanism drops MP3D SR from 40.1 to 36.5 for the LLaMA variant in the ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SG-Nav-GPT instead of OpenFMNav | zero-shot object navigation · MP3D | SR: 37.2 -> 40.2 | Helped |
| use SG-Nav-GPT instead of OpenFMNav | zero-shot object navigation · HM3D | SR: 52.5 -> 54.0 | Helped |
| use SG-Nav-GPT instead of OpenFMNav | zero-shot object navigation · RoboTHOR | SR: 44.1 -> 47.5 | Helped |
| add re-perception (ablation) | SG-Nav · MP3D | SR: 36.5 -> 40.1 | Helped |
| add re-perception (ablation) | SG-Nav · HM3D | SR: 49.6 -> 53.9 | Helped |
| add re-perception (ablation) | SG-Nav · RoboTHOR | SR: 41.9 -> 47.3 | Helped |
| add the scene graph and re-perception instead of removing both (ablation) | SG-Nav · MP3D | SR: 25.7 -> 40.1 | Helped |
| add a room node (ablation) | scene graph navigation · MP3D | SR: 39.4 -> 40.1 | Helped |
| add a group node (ablation) | scene graph navigation · MP3D | SR: 39.0 -> 40.1 | Helped |
| add scene graph edges (ablation) | SG-Nav · MP3D | SR: 38.0 -> 40.1 | Helped |
| add short edges (ablation) | SG-Nav · MP3D | SR: 39.0 -> 40.1 | Helped |
| add long edges (ablation) | SG-Nav · MP3D | SR: 39.5 -> 40.1 | Helped |
Surprising findings
- The LLaMA-7B variant performs nearly identically to the GPT-4 variant (40.1 vs 40.2 SR on MP3D), suggesting limited dependence on LLM choice.
- Group nodes contribute more than room nodes to performance (1.1 vs 0.7 SR drop on MP3D when removed).
Limitations
- Relies on a 2D vision-language model and vision foundation model to segment frames and merge masks, rather than an end-to-end 3D-aware online instance segmentation model.
- Only handles object-goal navigation; not evaluated on image-goal or vision-and-language navigation.
- Re-perception improves SR more than SPL since the agent still has to approach the goal to verify it, so path length is not reduced.
Experimental setup
Datasets: MP3D HM3D RoboTHOR
Benchmarks: MP3D HM3D RoboTHOR
Metrics: SR SPL SoftSPL
Baselines: SemEXP PONI ProcTHOR ZSON ProcTHOR-ZS CoW ESC L3MVN OpenFMNav VLFM
Builds on: Frontier-based Exploration ESC L3MVN VLFM
Closest papers by meaning
- UniGoal: Towards Universal Zero-shot Goal-oriented Navigation (CVPR 2025 · similarity 0.87)
- BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · similarity 0.87)
- GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance (NeurIPS 2024 · similarity 0.87)
- Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · similarity 0.86)
- RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation (NeurIPS 2026 · similarity 0.85)
- Navigating the Unseen: Zero-shot Scene Graph Generation via Capsule-Based Equivariant Features (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.