Research map / Vision-language and multimodal / Multimodal LLMs
XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces XLRS-Bench, a bilingual benchmark of 1,400 ultra-high-resolution remote sensing images (avg 8,500x8,500) with 16 sub-tasks for evaluating MLLM perception and reasoning, and finds that current general and RS-specific MLLMs perform poorly.
Problem
Existing remote sensing MLLM benchmarks use small image slices (about 512x512), rely on weakly verified annotations, and cover few perception and reasoning dimensions, so they do not test MLLMs on real ultra-high-resolution RS imagery.
Approach
A benchmark built on very large RS images (avg 8,500x8,500, many at 10,000x10,000) with mostly human-written VQA and grounding annotations, a semi-automated GPT-4o captioning pipeline with human refinement, 16 sub-tasks spanning 10 perception and 6 reasoning abilities, and English and Chinese versions.
Key results
- Qwen2-VL achieved the highest average VQA accuracy among evaluated models (41.10% Chinese, 38.99% English), while most models stayed below 50% on sub-tasks.
- GPT-4o scored below 25% accuracy on spatiotemporal reasoning (change detection counting), and visual grounding Acc@0.5 was at most 0.46 (GPT-4o, English) with Acc@0.7 at most 0.12 (InternVL2).
- Human accuracy exceeded 90% on the VQA sub-tasks, supporting benchmark validity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Qwen2-VL rather than GPT-4o on XLRS-Bench VQA | Qwen2-VL vs GPT-4o · XLRS-Bench-EN VQA | average accuracy: 38.99 vs 32.15 | Helped |
| Use Qwen2-VL rather than GPT-4o on XLRS-Bench VQA | Qwen2-VL vs GPT-4o · XLRS-Bench-ZH VQA | average accuracy: 41.10 vs 30.40 | Helped |
| Use higher-resolution input processing rather than a visual encoder with a lower image-size limit | MLLMs on XLRS-Bench · XLRS-Bench VQA | accuracy | Helped |
| Use GPT-4o rather than GeoChat for long-form English captioning | GPT-4o vs GeoChat · XLRS-Bench-EN detailed image captioning | BLEU-1: 16.74 -> 34.69 | Helped |
Surprising findings
- Models generally performed worse on perception than on reasoning tasks on XLRS-Bench, reversing the trend seen on common benchmarks.
- GPT-4o underperformed open-source models on spatiotemporal reasoning, with accuracy consistently below 25%.
Limitations
- Current MLLMs are limited to about 4K input, forcing compression of ultra-high-resolution images and losing small-object detail.
- Visual grounding performance is very low for all models, so the grounding task is near floor.
Experimental setup
Datasets: DOTA-v2 ITCVD MiniFrance Toronto Potsdam HRSCD VRSBench
Benchmarks: XLRS-Bench VRSBench
Metrics: accuracy Acc@0.5 Acc@0.7 BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE_L
Baselines: GPT-4o GPT-4o-mini Qwen2-VL LLaVA-OneVision LLaVA-Next LLaVA-1.5 CogVLM2 InternLM-XComposer-2.5 InternVL2 GeoChat
Builds on: LMMs-Eval MMBench MME-Realworld
Closest papers by meaning
- GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution (NeurIPS 2025 · similarity 0.91)
- VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.90)
- CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding (NeurIPS 2025 · similarity 0.88)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.87)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.