Research map / Multi-agent systems and games / Human-AI and goal-conditioned
CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a transformer urban navigation policy (CityWalker) on 2000+ hours of web city walking and driving videos with DPVO-derived action labels, then fine-tunes on 6 hours of teleoperated quadruped data.
Problem
Point-goal waypoint navigation in dynamic, map-free urban environments remains unsolved; expert teleoperation data is too small and lacks diversity, and VLM-based action labeling from videos is costly.
Approach
Derive action supervision from in-the-wild walking and driving videos using off-the-shelf visual odometry (DPVO) with per-trajectory step-length normalization, and train a DINOv2-based transformer with orientation loss and feature hallucination loss for imitation learning.
Key results
- Fine-tuned CityWalker reaches 87.8% arrival and 11.5° MAOE (scenario-mean 15.2°) on offline data, beating fine-tuned ViNT (70.7% arrival, 12.6° MAOE).
- Real-world success rate of 77.3% for fine-tuned CityWalker versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD.
- Zero-shot CityWalker surpasses fine-tuned ViNT in MAOE once trained on more than 1000 hours of walking videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CityWalker with expert data | CityWalker · Offline teleoperation test, All | MAOE (°): 12.7 -> 11.5 | Helped |
| Use CityWalker instead of ViNT fine-tuned | Real-world quadruped navigation · Real-world navigation, All | Success rate: 57.1 -> 77.3 | Helped |
| Add orientation loss (ablation) | CityWalker with 1000-hour walking data · Offline teleoperation test, scenario mean | MAOE (°): 17.03 -> 17.00 | No clear effect |
| Add feature hallucination loss (ablation) | CityWalker with 1000-hour walking data · Offline teleoperation test, scenario mean | MAOE (°): 17.00 -> 17.02 | No clear effect |
| Fine-tune with expert data (ablation) | CityWalker with 1000-hour walking data · Offline teleoperation test, scenario mean | MAOE (°): 17.02 -> 15.23 | Helped |
| Add feature hallucination loss to fine-tuned CityWalker (ablation) | CityWalker fine-tuned with expert data · Offline teleoperation test, scenario mean | MAOE (°): 15.23 -> 15.21 | No clear effect |
| Add orientation loss to fine-tuned CityWalker with feature hallucination loss (ablation) | CityWalker fine-tuned with expert data · Offline teleoperation test, scenario mean | MAOE (°): 15.21 -> 15.16 | No clear effect |
Surprising findings
- Feature hallucination loss hurt zero-shot performance, attributed to the human-to-quadruped domain and embodiment gap, though it helped after fine-tuning.
- A model trained only on driving videos performs similarly to zero-shot baselines, but mixing driving and walking data yields large gains.
Limitations
- Real-world deployment is sensitive to large GPS location noise from iPhone location services.
- Weak performance in detour scenarios, attributed to small proportion of such data in training videos.
- VO trajectories are noisy globally and suffer scale ambiguity, mitigated only by short-window relative poses.
Experimental setup
Datasets: in-the-wild city walking videos (2000+ hours) in-the-wild driving videos teleoperated expert data from Unitree Go1 (6 h fine-tune, 9 h test, NYC)
Benchmarks: offline teleoperation benchmark (L2, MAOE, arrival rate across critical scenarios) real-world navigation success rate on Unitree Go1
Metrics: L2 distance MAOE arrival rate navigation success rate AOE
Baselines: GNM ViNT NoMaD
Builds on: NoMaD ViNT GNM DPVO DINOv2
Closest papers by meaning
- CitySeeker: How Do VLMs Explore Embodied Urban Navigation with Implicit Human Needs? (ICLR 2026 · similarity 0.88)
- Navigation World Models (CVPR 2025 · similarity 0.86)
- RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation (CVPR 2025 · similarity 0.86)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.86)
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.85)
- Towards Autonomous Micromobility through Scalable Urban Simulation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.