Research map / Vision-language and multimodal / Human and embodied modeling
A scalar per patch from pre-trained ViTs enables fast moving navigation in the real world
Steeven Janny, Leonid Antsfeld, Christian Wolf
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a pre-trained distilled encoder (Dune) instead of a ResNet trained from scratch with RL | frozen ViT + CAP agent · HM3D val sim | SR (%): 53.5 -> 90.7 | Improved |
| initialize the agent from a Lidar-trained privileged agent and finetune on RGB | DinoV2 + CAP · HM3D val sim | SR (%): 63.4 -> 88.0 | Improved |
| initialize from the privileged Lidar agent | VC-1 + CAP · HM3D val sim | SR (%): 86.3 -> 90.3 | Improved |
| initialize from the privileged Lidar agent | Dune + CAP · HM3D val sim | SR (%): 90.7 -> 89.8 | No clear effect |
| initialize from the privileged Lidar agent | AM-Radio + CAP · real building | SR (%): 100 -> 92.9 | Hurt |
| replace CAP with PAP (value projection removed) | DinoV3 · HM3D val sim | SR (%): 0.0 -> 79.7 | Improved |
| add self-attention pooling projector in place of PAP or LPR (ablation) | DinoV3 · HM3D val sim | SR (%) | Hurt |
| increase read-out tokens from 4 to 8 in CAP (ablation) | DinoV2 + CAP · real building | SR (%): 97.6 -> 90.5 | Hurt |
| increase read-out tokens from 1 to 8 in PAP (ablation) | DinoV2 + PAP · HM3D val sim | SR (%): 81.7 -> 89.3 | Improved |
| replace CAP with PAP (K=4) (ablation) | DinoV2 · HM3D val sim | SR (%): 88.0 -> 86.3 | No clear effect |
| replace static PCA with learned linear patchwise reduction (ablation) | DinoV2 · HM3D val sim | SR (%): 51.5 -> 88.7 | Improved |
| increase LPR dimension from 4 to 8 (ablation) | DinoV3 + LPR · HM3D val sim | SR (%): 77.7 -> 31.6 | Hurt |
Closest papers by meaning
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.87)
- End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon (ICLR 2024 · similarity 0.87)
- Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction (ICLR 2024 · similarity 0.87)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.86)
- Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · similarity 0.86)
- STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
