Research map / Vision-language and multimodal / Robot action and navigation
Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenUAV, a realistic UE4/AirSim UAV VLN platform with a 12k-trajectory 6-DoF dataset, the assistant-guided UAV-Need-Help benchmark, and an MLLM (Vicuna-7B) hierarchical trajectory generator with backtracking DAgger data aggregation.
Problem
Existing UAV vision-language navigation benchmarks reuse ground-based VLN settings with discrete action sets, which ignore continuous 3D flight dynamics and the complexity of long, obstructed aerial search tasks.
Approach
Replace discrete UAV action sets with continuous 6-DoF trajectories collected on a realistic UE4+AirSim platform, add tiered assistant guidance to define a UAV object-search benchmark, and use an MLLM that predicts a long-range target pose plus a fine-grained trajectory decoder.
Key results
- On the Test Seen set with L1 assistant, Ours-DA reaches 17.45% SR and 15.76 SPL versus 8.37% SR for CMA; human operators reach 94.51% SR.
- On Test Unseen Object, Ours achieves 22.42% SR versus 9.06% for CMA.
- On Test Unseen Map, Ours achieves 4.18% SR versus 2.30% for CMA, showing limited zero-shot generalization to new maps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hierarchical MLLM-based trajectory generation instead of CMA | UAV-Need-Help, L1 assistant · Test Seen Full | SR: 8.37 -> 16.10 | Helped |
| add backtracking sampling-based data aggregation (ablation) | UAV Navigation LLM · Test Seen Full, L1 | SR: 16.10 -> 17.45 | Helped |
| add LoRA layers to the MLLM (ablation) | UAV Navigation LLM · UAV-Need-Help | OSR: 8% decrease when excluded | Helped |
| add LoRA layers to the MLLM (ablation) | UAV Navigation LLM · UAV-Need-Help | SR: 4% decrease when excluded | Helped |
| add cosine similarity loss for pose supervision (ablation) | UAV Navigation LLM · UAV-Need-Help | performance | Helped |
| add the learnable trajectory query (ablation) | UAV Navigation LLM · UAV-Need-Help | performance | Helped |
| reduce assistant guidance from L1 to L2 or L3 | UAV Navigation LLM · Test Seen Full | SR: 16.10 (L1) -> 12.98 (L2) -> 6.31 (L3) | Hurt |
| evaluate on unseen maps instead of unseen objects | UAV Navigation LLM · Test Unseen Map vs Test Unseen Object Full, L1 | SR: 22.42 -> 4.18 | Hurt |
Surprising findings
- Human operators with L1 assistance reach 94.51% SR, far above all automated methods, highlighting a large gap.
Limitations
- Absolute success rates remain low (best 17.45% SR on Test Seen), and performance drops sharply on unseen maps (4.18% SR).
- Relies on an external Grounding DINO detector for landing and on L1 assistant guidance that is close to ground truth.
Experimental setup
Datasets: OpenUAV target-oriented VLN dataset
Benchmarks: UAV-Need-Help
Metrics: SR OSR SPL NE
Baselines: Random Fixed Action CMA
Builds on: DAgger Cross-Modal Attention Grounding DINO AirSim
Closest papers by meaning
- SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery (NeurIPS 2026 · similarity 0.91)
- Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigation (CVPR 2026 · similarity 0.91)
- WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation (NeurIPS 2026 · similarity 0.91)
- UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning (NeurIPS 2025 · similarity 0.90)
- FlySearch: Exploring how vision-language models explore (NeurIPS 2025 · similarity 0.89)
- Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.