Research map / Vision-language and multimodal / Multimodal LLMs
FlySearch: Exploring how vision-language models explore
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FlySearch, a photorealistic Unreal Engine 5 UAV object-search benchmark that measures zero-shot exploration of VLMs, finding large gaps to humans especially on harder long-horizon search, and shows GRPO fine-tuning helps only partially.
Problem
Whether vision-language models can perform active, goal-driven exploration (object search with altitude control) in large, unstructured outdoor 3D environments remains untested.
Approach
A zero-shot outdoor UAV object-search benchmark built on Unreal Engine 5 with 3D (x, y, altitude) continuous actions, three difficulty tiers (FS-1, FS-Anomaly-1, FS-2), and human baselines, evaluating VLMs themselves rather than agent systems built on them.
Key results
- Best VLM Gemini 2.0 Flash reaches 42.0% on FS-1 versus 66.7% for untrained humans on City; on FS-2 the best VLMs score about 6% versus 60.8% for humans.
- GRPO fine-tuning of Qwen2.5-VL-7B on forest data raises FS-1 City success from 1.5% to 27.0% but leaves FS-2 at 0.0%.
- Compass-style actions drop Gemini's FS-1 forest success from 42.5% to 17.5%, and removing the grid overlay drops City success from 41.5% to 17%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-VL 7B with GRPO on Forest trajectories | Qwen2.5-VL 7B · FS-1 City | success rate (%): 1.5% to 21.5% | Helped |
| Fine-tune Qwen2.5-VL 7B with GRPO on Forest trajectories | Qwen2.5-VL 7B · FS-2 | success rate (%): 0.0% to 0.0% | No clear effect |
| Use supervised fine-tuning instead of the base model | Qwen2.5-VL 7B · FS-1 City | success rate (%): 1.5% to 11.1% | Helped |
| Replace Cartesian movements with compass commands (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-1 City and Forest | success rate (%): Gemini: 41.5% to 17.5% (City), 42.5% to 17.5% (Forest); Pixtral: 21.5% to 21% (City), 38% to 22% (Forest) | Hurt |
| Remove the grid overlay (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-1 City and Forest | success rate (%): Gemini: 41.5% to 17%; 42.5% to 31.5%; Pixtral: 21.5% to 15.5%; 38% to 20% | Hurt |
| Explicitly name the anomaly object to be found (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-Anomaly-1 Overall | success rate (%): Gemini: 35.5% to 46.5%; Pixtral: 15.0% to 20.5% | Helped |
| Reduce episode limit from 10 to 5 steps (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-1 Overall | success rate (%): Gemini: 42.0% to 38.0%; Pixtral: 29.8% to 24.5% | Hurt |
| Increase episode limit from 10 to 20 steps (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-1 Overall | success rate (%): Gemini: 42.0% to 39.5%; Pixtral: 29.8% to 24.8% | Hurt |
| Increase episode limit from 10 to 20 steps (ablation) | Gemini 2.0 Flash and Pixtral-Large · FS-1 Forest | success rate (%): Gemini: 42.5% to 45.5%; Pixtral: 38.0% to 36.0% | Mixed |
Surprising findings
- Increasing the step limit from 10 to 20 reduced performance for both Gemini and Pixtral on FS-1.
- Small open-weight models (below 11B) almost never succeed mainly because they fail to follow instructions, e.g. not issuing FOUND.
Limitations
- Only one fine-tuned model (Qwen2.5-VL-7B) and a simple prompting setup; no few-shot or prompt optimization tested.
- Human baseline is small (111 and 51 samples) and drawn from employed testers.
- Fine-tuning did not improve the harder FS-2 tier at all.
Experimental setup
Datasets: FlySearch forest environment scenarios (synthetic trajectories)
Benchmarks: FS-1 FS-Anomaly-1 FS-2
Metrics: success rate standard error steps ablation action space ablation grid overlay ablation
Baselines: GPT-4o Claude 3.5 Sonnet Gemini 2.0 Flash Pixtral-Large Qwen2-VL-72B Llava-Onevision 72B Qwen2.5-VL 7B InternVL-2.5 8B MPO Phi-3.5-vision Llava-Interleave-7b Human (untrained)
Builds on: UnrealCV Object-Goal Navigation GRPO LoRA
Closest papers by meaning
- Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology (ICLR 2025 · similarity 0.89)
- SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.88)
- Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method (CVPR 2025 · similarity 0.87)
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination (ICLR 2025 · similarity 0.87)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.87)
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.