Research map / Vision-language and multimodal / Human and embodied modeling
Learning Fine-Grained Alignment for Aerial Vision-Dialog Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the FG-AVDN entity-landmark dataset with a semi-automatic pipeline and proposes FELA, a semantic-grid navigator trained with grounding, captioning, and contrastive auxiliary tasks for aerial vision-dialog navigation.
Problem
Aerial vision-dialog navigation (ANDH/AVDN) needs drones to align dialog entities with landmarks in cluttered top-down views, but existing supervision is coarse and visual representations are condensed.
Approach
A precise semantic grid representation (visual features + xView object-category encoding + polar position encoding) plus three entity-landmark alignment auxiliary tasks (rotated bounding box prediction, landmark captioning, entity-landmark contrastive learning), supported by a new fine-grained dataset with entity-landmark annotations.
Key results
- FELA reaches 21.9% SR, 17.6% SPL, and 61.4 GP on the unseen testing split, versus 18.7% SR, 15.1% SPL, 56.5 GP for TG-GAT.
- Semantic grid representation alone (without alignment tasks) beats TG-GAT by 1.6% SR and 1.3% SPL on unseen validation.
- Removing the semantic encoding drops SPL to 13.5% versus 17.0% for visual-only grid, showing complementary visual and object encodings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FELA's semantic grid representation and alignment tasks | TG-GAT · ANDH unseen testing | SR: 18.7 -> 21.9 | Helped |
| add FELA's semantic grid representation and alignment tasks | TG-GAT · ANDH unseen testing | GP: 56.5 -> 61.4 | Helped |
| add entity-landmark aligning tasks (ablation) | FELA · ANDH unseen testing | SR: 20.3 -> 21.9 | Helped |
| add polar position encoding (ablation) | image + semantic grid representation · ANDH unseen validation | SR: 21.3 -> 23.9 | Helped |
| replace horizontal bounding-box prediction with rotated bounding-box prediction (ablation) | landmark grounding auxiliary task · ANDH unseen validation | SR: 20.6 -> 21.1 | Helped |
| add landmark semantic prediction to rotated-box prediction (ablation) | FELA alignment tasks · ANDH unseen validation | SPL: 17.9 -> 18.5 | Helped |
| add ELCL to LRBP and LSP (ablation) | FELA alignment tasks · ANDH unseen validation | SR: 22.6 -> 23.9 | Helped |
| use semantic encoding only instead of image encoding only (ablation) | grid representation · ANDH unseen validation | SPL: 17.0 -> 13.5 | Hurt |
| add image encoding to semantic encoding (ablation) | grid representation · ANDH unseen validation | SPL: 13.5 -> 18.3 | Helped |
| increase semantic grid scale from 5 × 5 to 7 × 7 (ablation) | FELA · ANDH unseen validation | SR: 20.2 -> 23.9 | Helped |
| increase semantic grid scale from 7 × 7 to 11 × 11 (ablation) | FELA · ANDH unseen validation | SR: 23.9 -> 21.5 | Hurt |
| train with extracted entity-landmark pairs (ablation) | FELA · ANDH unseen validation | SR: 20.1 -> 22.7 | Helped |
Surprising findings
- Semantic-only grid (object category encoding without visual features) performs worse than visual-only grid, attributed to predefined categories missing attribute-related landmarks.
Limitations
- Navigator performance remains unsatisfactory, so real-world drone deployment is deferred due to safety concerns.
- Fails on distance-related instructions (e.g., '80 meters') due to lack of absolute spatial awareness.
Experimental setup
Datasets: AVDN xView FG-AVDN
Benchmarks: ANDH (AVDN)
Metrics: SR SPL GP
Baselines: Transformer HAA-LSTM HAA-Transformer TA-GAT TG-GAT
Builds on: Episodic Transformer HAA-Transformer TG-GAT SAM LHRS-Bot RemoteCLIP
Closest papers by meaning
- LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigation (CVPR 2026 · similarity 0.88)
- Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory (CVPR 2026 · similarity 0.87)
- Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology (ICLR 2025 · similarity 0.85)
- SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery (NeurIPS 2026 · similarity 0.85)
- Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement (CVPR 2025 · similarity 0.84)
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.