Research map / Video / Video segmentation and tracking
NaviFormer: A Spatio-Temporal Context-Aware Transformer for Object Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NaviFormer, an encoder-decoder transformer that fuses spatial layout, temporal pose trajectory, and a passable frontier map for PPO-trained object navigation.
Problem
Object navigation agents using plain CNN fusion of spatial layouts and temporal pose trajectories miss long-range spatio-temporal dependencies, causing poor state representations and decision failures.
Approach
Builds a passable frontier map from occupancy and obstacle maps, encodes it with spatial layout and temporal pose features, and uses cross-attention decoders (spatial frontier and temporal pose) to produce frontier and target-related weights that reweight the agent's state representation before PPO-based long-term goal selection.
Key results
- NaviFormer reaches 82.6% SR on Gibson, 61.3% on HM3D, and 40.1% on MP3D val, beating SGM by 4.6/1.5/2.4 SR points.
- On Gibson, NaviFormer's SPL (40.9%) is below SGM's 44.0%, so it only matches average SPL there.
- Ablation: transformer-only encoder with frontier map (Φ4) underperforms the CNN variant Φ2 in SR on HM3D (58.7 vs 59.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SGM with NaviFormer | Object navigation agent · Gibson val | SR (%): 78.0 -> 82.6 | Helped |
| Replace SGM with NaviFormer | Object navigation agent · HM3D val | SR (%): 59.8 -> 61.3 | Helped |
| Replace SGM with NaviFormer | Object navigation agent · MP3D val | SR (%): 37.7 -> 40.1 | Helped |
| Replace SGM with NaviFormer | Object navigation agent · Gibson val | SPL (%): 44.0 -> 40.9 | Hurt |
| Replace SGM with NaviFormer | Object navigation agent · HM3D val | SPL (%): 29.4 -> 29.6 | Helped |
| Replace SGM with NaviFormer | Object navigation agent · MP3D val | SPL (%): 14.7 -> 15.1 | Helped |
| Add the passable frontier map to CNN encoding (ablation) | ObjectNav encoder · Gibson val | SR (%): 78.8 -> 79.7 | Helped |
| Add the passable frontier map to CNN encoding (ablation) | ObjectNav encoder · HM3D val | SR (%): 59.1 -> 59.8 | Helped |
| Add the passable frontier map to ViT encoding (ablation) | ObjectNav encoder · Gibson val | SR (%): 78.9 -> 79.5 | Helped |
| Add the passable frontier map to ViT encoding (ablation) | ObjectNav encoder · HM3D val | SR (%): 57.4 -> 58.7 | Helped |
| Add the spatial frontier decoder to the navigation encoder (ablation) | NaviFormer · Gibson val | SR (%): 78.4 -> 81.4 | Helped |
| Add the spatial frontier decoder to the navigation encoder (ablation) | NaviFormer · HM3D val | SR (%): 57.9 -> 60.7 | Helped |
Surprising findings
- A pure ViT encoder (Φ3) performed worse than a CNN encoder despite better value-loss convergence, suggesting transformers do not implicitly capture the needed long-range navigation dependencies.
Limitations
- Some baselines (Peanut, SGM) were re-implemented under the authors' setup, so comparisons may not be exact.
- Gibson SPL is lower than SGM's, and the paper notes the frontier emphasis can hurt efficiency.
Experimental setup
Datasets: Gibson HM3D MP3D
Benchmarks: Gibson HM3D MP3D
Metrics: SR SPL DTS FPS
Baselines: DD-PPO ANS SemExp L2M PONI Stubborn 3D-aware Peanut FSE SGM ESC L3mvn
Builds on: PPO SemExp PONI Fast Marching Method RedNet
Closest papers by meaning
- Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · similarity 0.86)
- C-NAV: Towards Self-Evolving Continual Object Navigation in Open World (NeurIPS 2025 · similarity 0.86)
- Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · similarity 0.86)
- Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction (ICLR 2024 · similarity 0.86)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.86)
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.