Research map / Vision-language and multimodal / Robot action and navigation
End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEBiT, a dual-encoder model that pre-trains a binocular ViT with cross-view completion and relative pose/visibility estimation, then trains an RL navigation policy for ImageNav and Instance-ImageNav.
Problem
End-to-end image-goal navigation (ImageNav and Instance-ImageNav) is hard because the perception module must learn a wide-baseline visual correspondence between an exemplar goal image and observations, which RL reward alone or standard auxiliary tasks fail to teach.
Approach
Use two pretext tasks as a prior: cross-view completion (CroCo) to pre-train a binocular ViT, then fine-tune it for extremely wide-baseline relative pose estimation plus a visibility prediction head, and feed its embedding to a recurrent RL policy alongside a monocular ResNet encoder.
Key results
- DEBiT-L (120M binocular params, frozen encoder, no adapters) reaches 82.0% SR and 59.6% SPL on ImageNav.
- Across capacity variants, pose accuracy at 1m&10deg rises from 80.3% (Tiny, 17M) to 97.5% (Large, 120M), while SR stays between 79.3% and 83.0%.
- The paper claims SOTA on ImageNav and Instance-ImageNav, with adapters giving further gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase binocular encoder capacity from Small to Base (ablation) | DEBiT · ImageNav Gibson-val | correct poses (1m & 10°): 82.7 -> 92.5 | Helped |
| Increase binocular encoder capacity from Small to Base (ablation) | DEBiT · ImageNav Gibson-val | success rate (%): 79.6 -> 83.0 | Helped |
| Increase binocular encoder capacity from Small to Base (ablation) | DEBiT · ImageNav Gibson-val | SPL (%): 52.1 -> 55.6 | Helped |
| Increase binocular encoder capacity from Base to Large (ablation) | DEBiT · ImageNav Gibson-val | correct poses (1m & 10°): 92.5 -> 97.5 | Helped |
| Increase binocular encoder capacity from Base to Large (ablation) | DEBiT · ImageNav Gibson-val | success rate (%): 83.0 -> 82.0 | Hurt |
| Increase binocular encoder capacity from Base to Large (ablation) | DEBiT · ImageNav Gibson-val | SPL (%): 55.6 -> 59.6 | Helped |
Surprising findings
- Higher binocular pose accuracy from larger encoders does not translate into higher success rate: DEBiT-B has the top SR (83.0%) while DEBiT-L has higher SPL (59.6%).
Limitations
- Evaluation is limited to simulated Gibson/Habitat environments without sim2real experiments.
- Frozen binocular encoder variant is used for most ablations; adapted variant needed for the best gains.
Experimental setup
Datasets: Gibson MP3D HM3D Habitat CroCo pre-training pairs
Benchmarks: ImageNav (Gibson val, 14 scenes) Instance-ImageNav
Metrics: success rate SPL percentage of correct poses at 1m/10deg and 2m/20deg visibility accuracy at ±0.05
Baselines: end-to-end ImageNav agents trained from reward modular map-based feature-matching method for Instance-ImageNav
Builds on: CroCo PPO AdaptFormer Vision Transformer
Closest papers by meaning
- Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation (CVPR 2024 · similarity 0.87)
- REGNav: Room Expert Guided Image-Goal Navigation (AAAI 2025 · similarity 0.86)
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.86)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.85)
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.