Research map / Detection and segmentation / Open-set and few-shot detection
Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Personalized Instance-based Navigation (PIN), a task where an agent must find a user-specific object instance among same-category distractors using visual and textual references, plus the PInNED dataset built on HM3D with Objaverse-XL objects, and benchmarks modular and end-to-end agents.
Problem
Object-driven navigation benchmarks target general categories or instances frozen in the scene at acquisition time, and do not test recognition of user-specific object instances that can be confused with same-category distractors and moved across locations.
Approach
Inject photo-realistic Objaverse-XL 3D objects into HM3D scenes at procedurally sampled surfaces, so personal objects can be placed and moved across episodes, and pair each target with visual reference renders and manually written textual descriptions, with same-category and other-category distractors.
Key results
- Best modular agent (DINOv2 matching) reaches 14.84 SR and 7.94 SPL on PInNED with same-category distractors, versus 9.14 SR for the best end-to-end agent (ZSON).
- Removing same-category distractors raises DINOv2 modular agent SR to 23.13, indicating the same-category distractors are the core difficulty.
- Keys, wallet and watch categories achieve 0% SR with the DINOv2 modular agent, showing small objects are hard to detect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove same-category distractors (ablation) | DINOv2 modular agent · PInNED | SR: 14.84 -> 23.13 | Helped |
| Remove same-category distractors (ablation) | DINOv2 modular agent · PInNED | SPL: 7.94 -> 11.61 | Helped |
| Remove same-category distractors (ablation) | ZSON · PInNED | SR: 9.14 -> 10.39 | Helped |
| Remove same-category distractors (ablation) | ZSON · PInNED | SPL: 7.18 -> 8.00 | Helped |
| Remove same-category distractors (ablation) | RIM (textual) · PInNED | SR: 7.12 -> 7.46 | Helped |
| Remove same-category distractors (ablation) | RIM (visual) · PInNED | SR: 8.80 -> 10.35 | Helped |
| Use visual rather than textual references | RIM · PInNED (same-category distractors) | SR: 7.12 -> 8.80 | Helped |
| Use visual rather than textual references | RIM · PInNED (without same-category distractors) | SR: 7.46 -> 10.35 | Helped |
Surprising findings
- End-to-end agents trained with imitation learning underperform modular agents, with ZSON at 9.14 SR versus 14.84 for DINOv2 modular.
- SuperGlue has a lower category error than DINOv2 and CLIP yet the worst SR and SPL among visual-reference methods.
Limitations
- Habitat rendering degrades texture quality and small object details, making small targets hard to detect at range.
- Textual references alone can be ambiguous and CLIP-based textual matching performs poorly on fine-grained attributes.
- End-to-end agents struggle with backtracking and recovery from wrong directions.
Experimental setup
Datasets: HM3D Objaverse-XL PInNED
Benchmarks: PInNED
Metrics: SR SPL CE D2G Steps %Match TM CM NM
Baselines: CLIP CLIP-Grad OWL SuperGlue IEVE PerSAM PerSAM-F DINO DINOv2 RIM ZSON
Builds on: ObjectNav InstanceImageNav GOAT-Bench IEVE ZSON RIM CoW Mod-IIN
Closest papers by meaning
- Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation (CVPR 2024 · similarity 0.87)
- Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation (CVPR 2024 · similarity 0.85)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.84)
- Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · similarity 0.84)
- MO-DDN: A Coarse-to-Fine Attribute-based Exploration Agent for Multi-Object Demand-driven Navigation (NeurIPS 2024 · similarity 0.84)
- C-NAV: Towards Self-Evolving Continual Object Navigation in Open World (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.