Research map / Vision-language and multimodal / Multimodal LLMs
GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SuperRS-VQA and HighRS-VQA (UHR remote-sensing VQA data) and GeoLLaVA-8K, a LLaVA-Next-based RS MLLM using background token pruning and anchored token selection to handle 8K inputs, reaching SOTA on XLRS-Bench.
Problem
Multimodal LLMs are limited to about 4K resolution and lack UHR remote-sensing training data, and ultra-high-resolution imagery produces a token explosion that makes training infeasible.
Approach
Prune semantically redundant background tokens via local clustering by semantic affinity, then retain informative object tokens using CLS-to-patch attention from the ViT (Anchored Token Selection), enabling 8K RS inputs at 24x compression, supported by new UHR VQA datasets built with manual annotation and influence-based selection.
Key results
- GeoLLaVA-8K (7B) reaches 51.5% average accuracy on XLRS-Bench, above Qwen2.5-VL-72B (50.2%) and all other evaluated open- and closed-source models.
- Adding high-resolution data raises LLaVA-Next-2K baseline average from 44.7 to 49.1; adding BTP+ATS reaches 51.5.
- On LRS-VQA (converted to MCQ), GeoLLaVA-8K scores 56.28% versus 55.07% for LLaVA-Next.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on SuperRS-VQA and HighRS-VQA with token optimization | LLaVA-Next-7B · XLRS-Bench | accuracy (Avg.): 44.7 -> 51.5 | Helped |
| Add high-resolution training data (ablation) | LLaVA-Next-2K · XLRS-Bench | accuracy (Avg.): 44.7 -> 49.1 | Helped |
| Add Background Token Pruning and Anchored Token Selection (ablation) | LLaVA-Next-2K with high-resolution data · XLRS-Bench | accuracy (Avg.): 49.1 -> 51.5 | Helped |
| Remove Anchored Token Selection while retaining Background Token Pruning (ablation) | LLaVA-Next-2K with high-resolution data · XLRS-Bench | accuracy (Avg.): 51.5 -> OOM | Hurt |
| Use 32× instead of 24× token compression (ablation) | GeoLLaVA-8K · XLRS-Bench | accuracy (Avg.): 51.5 -> 50.3 | Hurt |
| Use 16× instead of 24× token compression (ablation) | GeoLLaVA-8K · XLRS-Bench | accuracy (Avg.): 51.5 -> OOM | Hurt |
| Train on SuperRS-VQA only, without HighRS-VQA (ablation) | GeoLLaVA-8K training setup · XLRS-Bench | accuracy (Avg.): 49.4 -> 51.5 | Hurt |
| Train on SuperRS-VQA rather than VRSBench-train (ablation) | GeoLLaVA-8K training setup · XLRS-Bench | accuracy (Avg.): 34.8 -> 49.4 | Helped |
| Train using the proposed SuperRS-VQA and HighRS-VQA datasets (ablation) | GeoLLaVA-8K · XLRS-Bench | accuracy (Avg.): 34.8 -> 51.5 | Helped |
| Prune redundant background tokens (ablation) | LLaVA-Next-9K · XLRS-Bench | accuracy | Helped |
| Use GeoLLaVA-8K instead of LLaVA-Next | Remote-sensing MLLM evaluation · XLRS-Bench small-object subset | Overall accuracy: 34.1 -> 40.3 | Helped |
| Use GeoLLaVA-8K instead of LLaVA-Next | Remote-sensing MLLM evaluation · LRS-VQA | accuracy (%): 55.07 -> 56.28 | Helped |
Surprising findings
- Pruning 50% of background tokens on LLaVA-Next-9K improved accuracy over the unpruned version despite halving tokens.
- Training solely on VRSBench-train (123K low-resolution samples) yields much lower accuracy (34.8) than training on the 12K high-resolution SuperRS-VQA (49.4).
Limitations
- Focuses only on optical satellite imagery; SAR and multispectral modalities not evaluated.
- Official pilot experiments rely on a subset of 1,189 VQA pairs and 100 images, limiting statistical power.
Experimental setup
Datasets: SuperRS-VQA HighRS-VQA VRSBench-train GeoChat_Instruct DeepGlobe STAR FAIR1M 2.0 LoveDA DOTA
Benchmarks: XLRS-Bench LRS-VQA
Metrics: accuracy
Baselines: GeoChat GPT-4o GPT-4o-mini Claude 3.7 Sonnet Gemini 2.0 Flash Qwen2.5-VL-72B InternVL3-78B LLaVA-Next
Builds on: LLaVA-Next LLaVA-1.5 LESS logit lens
Closest papers by meaning
- XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery? (CVPR 2025 · similarity 0.91)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.91)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.90)
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery (AAAI 2026 · similarity 0.90)
- GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing (ICML 2025 · similarity 0.90)
- ZoomEarth: Active Perception for Ultra-High-Resolution Geospatial Vision-Language Tasks (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.