Research map / Vision-language and multimodal / Robot action and navigation
Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SimWorld-Robotics, a UE5-based simulator for procedurally generated photorealistic urban environments with pedestrians, traffic, and robots, plus two benchmarks (SimWorld-MMNav, SimWorld-MRS) and a 20K-step training set, showing that current VLMs struggle.
Problem
Existing embodied simulators focus on indoor/household or tabletop settings, and existing urban simulators lack photorealism, procedural scalability, and multi-agent/social dynamics needed to evaluate robots in large outdoor city environments.
Approach
A UE5 urban simulator combining procedural city generation, asynchronous control of humans, vehicles and robots, and rule-based pedestrian/traffic dynamics, used to build a multimodal instruction-following navigation benchmark and a two-robot search-and-meet benchmark with grounded communication.
Key results
- On SimWorld-MMNav easy, fine-tuned QwenVL 2.5 7B reaches 4.0% SR and 52.45% subtask SR, while zero-shot GPT-4o, Gemini 2.5 Flash and QwenVL 2.5 72B all score 0% SR.
- On SimWorld-MRS, GPT-4o with oracle planner reaches 65.00% CSR, while GPT-4o paired with RoCo drops to 33.33% CSR.
- On the hard MMNav setting, GPT-4o achieves 2.08% SR and Gemini 2.5 Flash has the fewest dynamic collisions (4.29) but more red-light violations (7.875).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune QwenVL 2.5 7B on SimWorld-20K | QwenVL 2.5 7B · SimWorld-MMNav easy | success rate (%): 0 -> 4.0 | Helped |
| Fine-tune QwenVL 2.5 7B on SimWorld-20K | QwenVL 2.5 7B · SimWorld-MMNav easy | subtask success rate (%): 16.86 -> 52.45 | Helped |
| Fine-tune QwenVL 2.5 7B on SimWorld-20K | QwenVL 2.5 7B · SimWorld-MMNav easy | distance progress (%): 7.82 -> 53.63 | Helped |
| Use reasoning models instead of zero-shot non-reasoning models | SimWorld-MMNav easy models · SimWorld-MMNav easy | success rate (%): 0 -> 5.0; 0 -> 8.3 | Helped |
| Use GPT-4o with RoCo instead of GPT-4o with an oracle planner | GPT-4o · SimWorld-MRS | collaborative success rate (%): 65.00 -> 33.33 | Hurt |
| Use GPT-4o with RoCo instead of GPT-4o with an oracle planner | GPT-4o · SimWorld-MRS | task progress (%): 76.90 -> 22.93 | Hurt |
| Use Gemini 2.5 Flash instead of GPT-4o | SimWorld-MMNav hard · SimWorld-MMNav hard | dynamic collisions: 10.37 -> 4.29 | Helped |
| Use Gemini 2.5 Flash instead of GPT-4o | SimWorld-MMNav hard · SimWorld-MMNav hard | red light violations: 3.02 -> 7.875 | Hurt |
| Add the explicit ReAct framework (ablation) | GPT-4o on SimWorld-MMNav · SimWorld-MMNav | navigation performance | Helped |
| Add segmentation observations (ablation) | GPT-4o on SimWorld-MMNav · SimWorld-MMNav | navigation performance | Helped |
Surprising findings
- Gemini 2.5 Flash's higher red-light violation count stems from freezing inside intersections after detecting red signals rather than failing to stop.
- Reasoning models (GPT-o3, GPT-o3-pro) are not suitable for real-time traffic avoidance due to inference latency.
Limitations
- Simulator focuses only on outdoor environments.
- Human agent action space is still limited.
- Fine-tuned model success rate remains low because training uses oracle action traces, limiting robustness.
Experimental setup
Datasets: SimWorld-20K
Benchmarks: SimWorld-MMNav SimWorld-MRS
Metrics: Success Rate Subtask Success Rate Distance Progress Static Collision Dynamic Collision Traffic Light Violation Collaborative Success Rate Task Progress
Baselines: GPT-4o GPT-o3 GPT-o3-pro Gemini 2.5 Flash QwenVL 2.5 7B QwenVL 2.5 72B Gemma 3 27B InternVL 3 78B HybridGPT VLA-RL RoCo
Builds on: SimWorld MetaUrban ReAct RoCo A*
Closest papers by meaning
- Towards Autonomous Micromobility through Scalable Urban Simulation (CVPR 2025 · similarity 0.89)
- Learning to Navigate Efficiently and Precisely in Real Environments (CVPR 2024 · similarity 0.87)
- Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation (CVPR 2025 · similarity 0.87)
- Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.86)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.