Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuRO, coupling a PICNN-based conformal uncertainty set with a robust optimization planner trained end-to-end, to improve data-scarce, partially observable multi-object visual navigation.
Problem
Visual navigation agents for multi-object navigation (MultiON) overfit under data scarcity and generalize poorly; integrating learned predictions with downstream optimization is unstable because prediction errors propagate and partial observability hinders global optimization.
Approach
Use PICNNs with conformal calibration to map noisy visual predictions into convex uncertainty sets that parameterize a robust pursuit-evasion optimization planner; optimization output provides an action offset and a reward signal that is backpropagated (via KKT implicit differentiation) into the perception network.
Key results
- On S-MON m=2 test, NeuRO reaches 80 Success, 86 Progress, 66 SPL, versus 76/84/60 for the best prior HTP-GCN/Lyon entries.
- On S-MON m=3 test, NeuRO reaches 62 Success and 40 SPL, against 57 Success and 36 SPL for Lyon.
- Under cross-m transfer, NeuRO's average drop is smaller than OracleEgoMap and Lyon (e.g., train m=1 to eval m=2: -12% vs -17% for OracleEgoMap and -15% for Lyon).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply NeuRO instead of Lyon | S-MON · S-MON m=2 Test | Success: 76 -> 80 | Helped |
| apply NeuRO instead of Lyon | S-MON · S-MON m=3 Test | Success: 57 -> 62 | Helped |
| apply NeuRO instead of OracleEgoMap | U-MON · U-MON m=2 Test | Success: 54 -> 62 | Helped |
| apply NeuRO instead of OracleEgoMap | U-MON · U-MON m=3 Test | Success: 36 -> 45 | Helped |
| increase α from 0.01 to 0.20 (ablation) | NeuRO · S-MON m=2 | Success: 80 -> 66 | Hurt |
| increase α from 0.01 to 0.20 (ablation) | NeuRO · S-MON m=2 | Prediction Error Mean: 0.90 -> 0.76 | Hurt |
| increase action blending factor λ from 0.05 to 0.20 (ablation) | NeuRO · S-MON m=2 | Success: 67 -> 80 | Helped |
| increase action blending factor λ from 0.20 to 0.40 (ablation) | NeuRO · S-MON m=2 | Success: 80 -> 52 | Hurt |
| increase optimization scale from E=3, τ=4 to E=5, τ=6 (ablation) | NeuRO · S-MON m=2 | Success: 75 -> 80 | Helped |
| replace task-specific S-MON* optimization with S-MON# formulation (ablation) | NeuRO · S-MON m=2 | Success: 80 -> 76 | Hurt |
| replace task-specific S-MON* optimization with S-MON# formulation (ablation) | NeuRO · S-MON m=2 | SPL: 66 -> 60 | Hurt |
| increase basis expansion grid size from E=20 to E=100 (ablation) | NeuRO with basis function expansion · S-MON m=2 | Success: 82 -> 85 | No clear effect |
Surprising findings
- Increasing the task weight lambda beyond 0.4 causes about a 20% drop in Success, showing excessive reliance on the optimization action hurts learning.
- Scaling the optimization grid via basis-function expansion gave no substantial navigation improvement at larger E, despite lower solve time scaling.
Limitations
- Relies on convex approximations of uncertainty via PICNNs, which cannot fully capture non-convex uncertainty.
- Optimization complexity grows as O(E^2 tau), limiting grid scale.
- Gains reported on a discrete grid abstraction with limited field of view; larger-scale results deferred to appendix show no navigation gain.
Experimental setup
Datasets: MultiON-style simulated navigation episodes (U-MON and S-MON)
Benchmarks: S-MON m=2 S-MON m=3 U-MON m=2 U-MON m=3
Metrics: Success Progress SPL PPL
Baselines: SMT FRMQN ProjNeuralMap ObjRecogMap OracleEgoMap OracleMap Lyon HTP-GCN
Builds on: MultiON task-based end-to-end learning partially input convex neural networks goal vector method
Closest papers by meaning
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.88)
- Selective Visual Representations Improve Convergence and Generalization for Embodied AI (ICLR 2024 · similarity 0.87)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
- STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization (NeurIPS 2025 · similarity 0.86)
- Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · similarity 0.86)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.