Research map / 3D vision / 3D generation
Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a latent navigability representation by training a blind auxiliary "mole" agent to navigate to subgoals from waypoints using only the main agent's latent state, improving sim2real and noisy-sim transfer for PointGoal navigation.
Problem
Learning actionable, transferable spatial representations for visual navigation that avoid explicit reconstruction and transfer robustly under the sim2real gap.
Approach
A blind auxiliary agent (the "mole") navigates short sub-episodes to subgoals using only the main agent's recurrent representation at waypoints, with behavior-cloning gradients flowing back into the main representation; blindness forces the representation to hold navigable map information rather than reactive cues.
Key results
- The hybrid Navigability+BC agent reaches 95.5 Success / 80.3 SPL in Eval Sim and 90.9 Success / 73.3 SPL in NoisySim, versus 92.0/79.6 for pure BC and 74.6/55.8 for PPO in NoisySim.
- On a real Locobot over 11 episodes, Navigability+BC achieves 81.8 Success / 45.6 SPL / 44.0 sSPL, versus 72.7 Success for BC and 45.5 for PPO.
- Navigability-only training sees only about 5% (2.5M) of visual observations yet is competitive in simulation (92.9 Success) and outperforms BC in noisy settings (86.8 vs 76.0 Success).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Navigability pretraining to long-episode BC | PointGoal agent · PointGoal test, NoisySim | Success: 89.6 -> 90.9 | Helped |
| Add Navigability pretraining to long-episode BC | PointGoal agent · PointGoal test, Simulation | Success: 94.2 -> 95.5 | Helped |
| Use Navigability + BC instead of BC on the real robot | Locobot PointGoal agent · Real office, 11 episodes | Success: 72.7 -> 81.8 | Helped |
| Use Navigability + BC instead of map reconstruction + BC | PointGoal agent · PointGoal test, NoisySim | Success: 76.7 -> 90.9 | Helped |
| Add Navigability pretraining to PPO | PointGoal agent · PointGoal test, NoisySim | Success: 74.6 -> 85.2 | Helped |
| Add PPO training to Navigability | PointGoal agent · PointGoal test, Simulation | Success: 92.9 -> 91.5 | Hurt |
| Set the main hidden state to zero at transitions to short episodes (ablation) | Navigability + BC · PointGoal test, Simulation | Success: 94.2 -> 90.3 | Hurt |
| Restore the main hidden state from the last waypoint at transitions to short episodes (ablation) | Navigability + BC · PointGoal test, Simulation | Success: 94.2 -> 88.6 | Hurt |
| Initialize the blind policy hidden state with the representation rather than feed it as an observation (ablation) | Navigability + BC · PointGoal test, Simulation | Success: 92.8 -> 95.5 | Helped |
| Initialize the blind policy hidden state with the representation rather than feed it as an observation (ablation) | Navigability + BC · PointGoal test, NoisySim | Success: 90.9 -> 80.7 | Hurt |
| Add 128 dimensions to the blind policy hidden state (ablation) | Navigability + BC · PointGoal test, Simulation | Success: 95.5 -> 90.4 | Hurt |
| Add 128 dimensions to the blind policy hidden state (ablation) | Navigability + BC · PointGoal test, NoisySim | Success: 80.7 -> 83.3 | Helped |
Surprising findings
- Navigability-only training, which sees only about 5% of visual observations, remains competitive with PPO and BC in simulation.
- Always continuing the hidden state across teleports between short and long episodes performed best, suggesting data volume matters more than state continuity.
Limitations
- Real-robot evaluation limited to 11 episodes in one office environment.
- Sim2real results are not directly comparable to other methods due to differing setups.
- Shortest-path GT and privileged simulator information are needed to generate blind-agent training signals.
Experimental setup
Datasets: Gibson
Benchmarks: Habitat PointGoal (Gibson val/test) NoisySim (simulated noise) real Locobot office robot episodes
Metrics: Success SPL Soft-SPL IoU Sym-SPL
Baselines: PPO behavior cloning AUX reconstruction + BC Map+Plan (classical) data augmentation BC
Builds on: PPO behavior cloning Habitat Gibson dataset
Closest papers by meaning
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.89)
- Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · similarity 0.87)
- Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments (ICML 2024 · similarity 0.86)
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.86)
- Versatile Navigation Under Partial Observability via Value-guided Diffusion Policy (CVPR 2024 · similarity 0.86)
- Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.