Research map / Vision-language and multimodal / Visual question answering
EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EarthVQA, a 6000-image remote sensing VQA dataset with 208,593 relational reasoning QA pairs, and SOBA, an object-centric VQA framework with a numerical difference loss for counting.
Problem
Remote sensing VQA mostly uses auto-generated simple counting/judging questions and global feature fusion, lacking relational reasoning over geospatial objects needed for city planning.
Approach
Object-centric VQA: segmentation-derived pixel-level visual prompts and pseudo masks feed object-guided attention, a visual self-attention encoder, and bidirectional cross-attention between vision and language; a numerical difference loss adds distance-based penalties to cross-entropy to unify classification and counting.
Key results
- SOBA achieves 78.14% overall accuracy (OA) and 0.839 OR on EarthVQA test, outperforming the best general method Instruct-BLIP (75.25% OA) and remote sensing RSIVQA (73.71% OA).
- The numerical difference loss lowers relational counting RMSE to 1.1457 from 1.2417 for BAN-style CE baselines, and ND loss adds gains of 0.6 OA points over CE in the ablation (77.54 to 78.14).
- Bidirectional cross-attention (VVV-LLL ordering) improves OA to 78.14 versus 77.11-77.57 for one-way or other orderings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOBA rather than MCAN | EarthVQA VQA · EarthVQA test | OA (%): 77.07 -> 78.14 | Helped |
| Add BCA to VSA (ablation) | SOBA · EarthVQA | OA (%): 72.55 -> 74.91 | Helped |
| Add semantic prompts to VSA + BCA (ablation) | SOBA · EarthVQA | OA (%): 74.91 -> 77.30 | Helped |
| Add OGA to VSA + BCA + prompts (ablation) | SOBA · EarthVQA | OA (%): 77.30 -> 77.54 | Helped |
| Add ND loss to VSA + BCA + prompts + OGA (ablation) | SOBA · EarthVQA | OA (%): 77.54 -> 78.14 | Helped |
| Use channel-based OGA instead of concatenation alone (ablation) | SOBA object guidance · EarthVQA | OA (%): 77.61 -> 78.14 | Helped |
| Use bidirectional VVV-LLL cross-attention instead of one-way VVVVVV (ablation) | SOBA · EarthVQA | OA (%): 77.53 -> 78.14 | Helped |
| Replace ND loss with CE loss (ablation) | SOBA · EarthVQA | OA (%): 78.14 -> 77.54 | Hurt |
| Replace ND loss with DIW (ablation) | SOBA · EarthVQA | OA (%): 78.14 -> 77.38 | Hurt |
| Replace ND loss with OHEM (ablation) | SOBA · EarthVQA | OA (%): 78.14 -> 77.85 | Hurt |
| Replace LSTM with BERT-Base (ablation) | Swin-T encoder variant · EarthVQA | OA (%): 77.22 -> 77.63 | Helped |
| Increase the encoder from Swin-T to Swin-S (ablation) | SOBA encoder · EarthVQA | OA (%): 77.22 -> 78.01 | Helped |
Surprising findings
- Channel attention variants gave more stable gains than spatial attention for fusing object masks with visual prompts.
- Focal loss, DIW, and other imbalance-focused losses do not help regression-style counting, motivating the ND loss.
Limitations
- Segmentation and VQA are trained in two separate stages without end-to-end interaction.
- Answer distribution is imbalanced (top 30 of 166 answers), which poses challenges.
- Only evaluated on urban and rural scenes from three Chinese cities (Nanjing, Changzhou, Wuhan).
Experimental setup
Datasets: EarthVQA LoveDA
Benchmarks: EarthVQA
Metrics: accuracy OA RMSE OR mIoU
Baselines: SAN MAC BUTD BAN MCAN D-VQA BLIP-2 Instruct-BLIP RSVQA RSIVQA
Builds on: LoveDA MCAN BUTD BLIP-2 Semantic-FPN
Closest papers by meaning
- VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.89)
- TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.88)
- See What We Cannot See: A Geo-guided Reasoning Benchmark for Object Counting under Adverse Earth Observation Conditions (CVPR 2026 · similarity 0.88)
- InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition (NeurIPS 2025 · similarity 0.88)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.88)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.