Research map / Large language models / Agents, code and math
EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EfficientNav, an on-device zero-shot object-goal navigation system that uses discrete KV-cache memory caching, attention-based memory clustering, and CLIP-based semantics-aware memory retrieval to let small LLaVA/LLaMA planners match or beat GPT-4-based planners with much lower latency.
Problem
Zero-shot LLM-based object-goal navigation relies on giant cloud LLMs such as GPT-4; small on-device LLMs lose success rate on complex navigation maps, and growing map prompts create KV-cache memory limits and high planning latency on local devices.
Approach
Cluster navigation-map objects into groups whose KV caches are precomputed independently so any subset can be loaded and reused regardless of order; cluster new objects using LLM attention from a few layers; and select relevant groups with a CLIP-scored knapsack under a device memory budget to prune redundant context for small LLMs.
Key results
- EfficientNav-34b reaches 80.0% SR and 41.5 SPL on HM3D versus 68.9 SR for the GPT-4-based LFG, an 11.1-point SR gain.
- Compared with the GPT-4 planner, EfficientNav-11b reduces real-time latency from 5.80s to 0.35s (6.7x) and end-to-end latency from 59.34s to 12.70s (4.7x) on an A6000.
- Ablation on LLaVA-34b: discrete memory caching alone lowers real-time latency from 5.63s to 2.42s with SR nearly unchanged (43.1 vs 42.7), while attention-based clustering raises SR to 63.3 and semantics-aware retrieval to 80.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EfficientNav instead of LFG | HM3D ObjNav · HM3D | SR: 11.1% | Helped |
| use EfficientNav instead of GPT-4 Planner | HM3D ObjNav on A6000 · HM3D | real-time latency: 6.7× | Helped |
| use EfficientNav instead of GPT-4 Planner | HM3D ObjNav on A6000 · HM3D | end-to-end latency: 4.7× | Helped |
| add discrete memory caching (ablation) | LLaVA Planner-34b · HM3D | real-time latency: 5.63s -> 2.42s | Helped |
| add discrete memory caching (ablation) | LLaVA Planner-34b · HM3D | SR: 42.7 -> 43.1 | No clear effect |
| add attention-based memory clustering (ablation) | LLaVA Planner-34b with discrete memory caching · HM3D | SR: 43.1 -> 63.3 | Helped |
| add attention-based memory clustering (ablation) | LLaVA Planner-34b with discrete memory caching · HM3D | SPL: 21.0 -> 34.2 | Helped |
| add semantics-aware memory retrieval (ablation) | LLaVA Planner-34b with attention-based memory clustering · HM3D | SR: 63.3 -> 80.0 | Helped |
| increase KV-cache memory budget from 16GB to 40GB (ablation) | EfficientNav with LLaVA-34b · HM3D | SR: 74.7 -> 80.3 | Helped |
| increase KV-cache memory budget from 16GB to 40GB (ablation) | EfficientNav with LLaVA-34b · HM3D | real-time latency: 0.59s -> 0.93s | Hurt |
| replace CLIP group selection with MPNet (ablation) | EfficientNav with LLaVA-34b · HM3D | SR: 80.0 -> 79.3 | No clear effect |
| replace CLIP group selection with MiniLM (ablation) | EfficientNav with LLaVA-34b · HM3D | SR: 80.0 -> 80.3 | No clear effect |
Surprising findings
- Adaptive sparse attention methods (Minference, FlexPrefill) drop SR to about 35-37% with LLaVA-34b, far below full-attention EfficientNav, because they minimize the gap to full attention rather than targeting retrieval.
- Replacing the CLIP group selector with an LLM gives only marginally higher SR (80.7 vs 80.0) at much higher latency (6.55s vs 0.87s).
Limitations
- Small LLM inference still slower than small specialized models, so unsuitable for extremely low-latency applications.
- Does not address cloud communication latency or privacy concerns of cloud serving.
- Evaluated only on HM3D with Habitat; the zero-shot approach may lack specialized knowledge for high-risk settings.
Experimental setup
Datasets: HM3D
Benchmarks: HM3D
Metrics: success rate SPL real-time latency end-to-end latency cache hit rate
Baselines: LFG InstructNav PixelNav VoroNav L3MVN OVRL ESC SemExp DD-PPO ZSON Habitat-web GPT-4 planner
Builds on: LLaVA Grounding DINO CLIP vLLM
Closest papers by meaning
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.89)
- C-NAV: Towards Self-Evolving Continual Object Navigation in Open World (NeurIPS 2025 · similarity 0.86)
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.85)
- Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · similarity 0.85)
- Trajectory Diffusion for ObjectGoal Navigation (NeurIPS 2024 · similarity 0.85)
- BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.