Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning to Navigate Efficiently and Precisely in Real Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an end-to-end recurrent PointGoal navigation agent in Habitat with a second-order identified motion model and sensing noise, and transfers it zero-shot to a real differential-drive robot.
Problem
End-to-end visual navigation policies trained in photorealistic simulators suffer a large sim2real gap in actuation (idealized teleport-style motion) and sensing, leading to inefficient and unreliable real-robot navigation.
Approach
Predict discretized linear/angular velocity commands asynchronously while the robot moves, and model the robot plus its low-level PID controller as an identified asymmetric second-order dynamical system inside Habitat, together with noise models for odometry and AMCL localization, so the agent learns to anticipate motion.
Key results
- With the identified dynamical model, the velocity agent reaches 97.4% SR and 82.2% SPL on HM3D/2.5k in simulation, versus 0.8% SR without the motion model.
- On the real robot (Office/20, 20 episodes), the proposed agent achieves 40.0% SR and 28.9% SPL versus 20.0% SR for the velocity agent without a dynamical model and 15.0% for the position-discrete agent.
- In the real office, the finetuned agent reaches 55.0% SR vs. 80.0% SR for the ROS 2 NavStack planner, which uses a prebuilt occupancy map.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add identified second-order dynamical motion model in simulation training (vs. constant-velocity sim) (ablation) | velocity disc(28) agent · Sim (+dyn.) HM3D/2.5k | SR%: 0.8 -> 97.4 | Helped |
| add identified second-order dynamical motion model in simulation training (ablation) | velocity disc(28) agent · Real Office/20 | SR%: 20.0 -> 40.0 | Helped |
| velocity action space with dynamical model vs position discrete actions [43] | end-to-end agent · Real Office/20 | SR%: 15.0 -> 40.0 | Helped |
| finetune agent with RL on Matterport scan of the building | proposed agent vs ROS 2 map-based planner · Real Office/20 | SR%: 80.0 (ROS 2) vs 55.0 (ours finetuned) | Hurt |
| remove auxiliary goal-compass supervision from static goal agent (ablation) | static point goal g0 agent · Real Office/20 | SR%: 50.0 -> 50.0 (with supervision 40.0 in Table 1 variant; unsupervised 50.0) | No clear effect |
| remove absolute AMCL localization input from static goal agent (ablation) | static point goal g0 agent with supervision · Real Office/20 | SR%: 40.0 -> 25.0 | Hurt |
| use dynamic point goal gt from external localization instead of static g0 (ablation) | agent with localization · Real Office/20 | SR%: 50.0 (g0) -> 35.0 (gt) | Hurt |
| add reset of recurrent state every 10 seconds at real deployment | real robot agent · Real Office/20 | SR% | No clear effect |
Surprising findings
- Removing the auxiliary goal supervision did not hurt in simulation (100% vs 70% SR in one setting), and the unsupervised static-goal variant performed well.
- Agents trained without the motion model achieved high success in their own training setting (98.0% SR) yet collapsed to 0.8% SR under realistic motion, showing that high training-domain performance is not indicative of real performance.
Limitations
- Real-robot evaluation limited to 20 episodes in one office building.
- Real-world performance remains well below the ROS 2 map-based planner and the authors note room for improvement.
- Agent can get discouraged by blocked passages due to sim2real mismatch in recurrent memory; mitigated by resetting memory every 10 seconds.
Experimental setup
Datasets: HM3D
Benchmarks: HM3D/2.5k (simulation) Office/20 (real and simulation)
Metrics: success rate SPL SCT
Baselines: Position discrete(4) agent (Chattopadhyay/Partsey-style) Yokoyama et al. velocity agent ROS 2 NavStack
Builds on: PPO Habitat AMCL SCT metric
Closest papers by meaning
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.89)
- Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration (NeurIPS 2025 · similarity 0.87)
- Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation (CVPR 2025 · similarity 0.87)
- Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation (CVPR 2024 · similarity 0.87)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.86)
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.