Research map / Vision-language and multimodal / Multimodal LLMs
GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeoPixel, an end-to-end high-resolution remote sensing LMM with pixel-level grounding via SAM-2 mask decoding, plus the GeoPixelD grounded conversation dataset built with set-of-marks prompting and spatial priors.
Problem
Existing large multimodal models perform poorly on remote sensing imagery due to overhead viewpoint, scale variation, small objects and low input resolution, and lack pixel-level grounding and grounded conversation data for RS.
Approach
Combine adaptive high-resolution image partitioning (IXC-2.5 style) with a [SEG] token decoded into masks by a SAM-2 grounding encoder and pixel decoder, and build a RS grounded-conversation dataset using prior-informed set-of-marks prompting and LLM paraphrasing with text marking.
Key results
- On RS-GCG overall, GeoPixel achieves CIDEr 21.6 (uni-target) and overall AP50 19.0, mIoU 52.3, Recall 38.8, outperforming GLaMM-ft (overall mIoU 46.4, AP50 12.5).
- On RRSIS-D test, GeoPixel-ft reaches P@0.5 83.33, oIoU 84.90, mIoU 67.30, above the prior best RMSIN (74.26 / 77.79 / 64.20).
- Increasing inference patches from P=1 to P=9 raises CIDEr from 14.6 to 20.5 and mIoU from 47.8 to 52.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GeoPixel instead of GLaMM-ft | RS-GCG · RS-GCG Overall | CIDEr: 15.7 -> 21.6 | Helped |
| Use GeoPixel instead of GLaMM-ft | RS-GCG · RS-GCG Overall | mIoU: 46.4 -> 52.3 | Helped |
| Fine-tune GeoPixel on RRSIS-D | GeoPixel · RRSIS-D test | P@0.5: 74.26 -> 83.33 | Helped |
| Increase inference patches from 1 to 9 (ablation) | GeoPixel trained with 9 patches · RS-GCG Overall | CIDEr: 14.6 -> 20.5 | Helped |
| Increase inference patches from 1 to 9 (ablation) | GeoPixel trained with 9 patches · RS-GCG Overall | METEOR: 23.1 -> 24.3 | Helped |
| Add Set-1B complex annotations to Set-1A (ablation) | GeoPixel with trainable vision projection · RS-GCG data complexity | mIoU: 48.0 -> 51.7 | Helped |
| Train rather than freeze the vision projection layer (ablation) | GeoPixel trained on Set-1A and Set-1B · RS-GCG | AP50: 15.3 -> 17.8 | Helped |
| Use instance-only rather than mixed annotations (ablation) | GeoPixel · RS-GCG annotation complexity | Recall: 33.3 -> 48.8 | Helped |
Surprising findings
- Mixed instance-and-semantic annotations gave the lowest mask recall (33.3) in the annotation complexity ablation, below instances-only (48.8) and semantic-only (37.7).
Limitations
- Erroneous or fragmented masks when deciding object presence, quantity, and instance vs. semantic mask type.
- Confuses instance masks at the same spatial location in dense or crowded scenes.
- Manual curation of test set required about 350 man-hours; training data relies on automated LMM-generated annotations.
Experimental setup
Datasets: iSAID GeoPixelD RRSIS-D
Benchmarks: RS-GCG (GeoPixelD test) RRSIS-D
Metrics: CIDEr METEOR AP50 mIoU Recall P@0.5 oIoU
Baselines: GLaMM LISA PixelLM GLaMM-ft RMSIN LAVT LGCE BRINet
Builds on: LISA GLaMM InternLM-XComposer-2.5 SAM-2 set-of-marks prompting
Closest papers by meaning
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.92)
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding (CVPR 2026 · similarity 0.90)
- GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution (NeurIPS 2025 · similarity 0.90)
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.88)
- EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues (CVPR 2025 · similarity 0.87)
- TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.