Research map / Vision-language and multimodal / Visual question answering
VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VHM, a remote sensing VLM trained on a 1.4M-image rich-content caption dataset (VersaD) and an honest instruction dataset (HnstD) with deceptive questions, adding multi-level visual features on LLaVA/CLIP-L/Vicuna-7B.
Problem
Existing remote sensing VLMs are pretrained on sparse-content captions that mention only a few prominent objects, and fine-tuned only on factual questions, so they lack detailed understanding and tend to affirm nonexistent objects (dishonesty).
Approach
Pretrain on rich-content captions generated by an off-the-shelf model (Gemini-Vision) covering image properties, object attributes and scene context; add an honesty instruction dataset mixing factual and deceptive questions; fuse multi-level CLIP features (layers 8, 16, 24) into the projector.
Key results
- VHM reaches 85.13% average scene classification accuracy across five datasets, versus 80.17% for LHRS-Bot.
- On DIOR-RSVG visual grounding, VHM scores 56.17 vs 44.58 for CogVLM, a gain of 11.59 points.
- On HnstD, VHM reaches 85.06% on presence and 90.67% on deceptive absolute-position questions, far above baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VersaD RS pretraining (ablation) | VHM with single-level visual features · Scene classification average | accuracy: 73.22 -> 84.71 | Helped |
| add VersaD RS pretraining (ablation) | VHM with single-level visual features · Visual Question Answering | score: 77.19 -> 82.09 | Helped |
| add VersaD RS pretraining (ablation) | VHM with single-level visual features · Visual Grounding | score: 28.23 -> 51.06 | Helped |
| add VersaD RS pretraining (ablation) | VHM with single-level visual features · Honest Question Answering | score: 61.65 -> 80.50 | Helped |
| pretrain on VersaD instead of SkyScript (ablation) | VHM with multi-level visual features · Visual Grounding | score: 16.01 -> 59.52 | Helped |
| pretrain on VersaD instead of RS5M-5M (ablation) | VHM with multi-level visual features · Visual Grounding | score: 48.71 -> 59.52 | Helped |
| pretrain on VersaD instead of RS5M-1.5M (ablation) | VHM with multi-level visual features · Visual Grounding | score: 48.08 -> 59.52 | Helped |
| use multi-level visual features instead of single-level final-layer features (ablation) | VHM · DIOR-RSVG | visual grounding score: 46.96 -> 56.17 | Helped |
| use multi-level visual features instead of single-level final-layer features (ablation) | VHM · Scene classification | score: 85.47 -> 85.13 | Hurt |
| use multi-level visual features instead of single-level final-layer features (ablation) | VHM · Visual Question Answering | score: 81.74 -> 81.54 | Hurt |
| use multi-level visual features instead of single-level final-layer features (ablation) | VHM · Honest Question Answering | score: 79.03 -> 80.93 | Helped |
| add VersaD pretraining instead of relying on the LLaVA pretrained weights (ablation) | VHM · Scene classification average | accuracy: 73.22 -> 84.71 | Helped |
Surprising findings
- VersaD pretraining with 82.3% caption accuracy outperforms SkyScript (96.1% accuracy) on visual grounding by over 43%, suggesting rich content compensates for label noise.
Limitations
- Lacks pixel-wise perception, so it cannot do semantic segmentation or change detection.
- Weak on object counting (MAE 6.75) and geometric measurement (MAE 12.82).
- Relative-position honesty accuracy remains limited (47.52% factual), and VersaD captions contain about 18% inaccurate sentences.
Experimental setup
Datasets: Million-AID CrowdAI fMoW CVUSA CVACT LoveDA DOTA-v2 FAIR1M DIOR RSVQA-LR RSVQA-HR DIOR-RSVG
Benchmarks: NWPU METER-ML SIRI-WHU AID WHU-RS19 RSVQA-LR RSVQA-HR DIOR-RSVG HnstD DOTA-v2 CrowdAI GID
Metrics: accuracy Acc@0.5 IoU mean absolute error complexity-aware IoU F1-measure
Baselines: LLaVA-1.5 MiniGPTv2 Qwen-VL-Chat Gemini-Vision LHRS-Bot RSGPT GeoChat SkyEyeGPT EarthGPT CogVLM
Builds on: LLaVA CLIP Vicuna-v1.5
Closest papers by meaning
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.91)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.90)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.89)
- Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models (NeurIPS 2025 · similarity 0.89)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.89)
- VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.