Research map / Vision-language and multimodal / Multimodal LLMs
LISAt: Language-Instructed Segmentation Assistant for Satellite Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LISAt, a geospatial vision-language model that adds LISA-style [SEG]-token mask decoding to a RemoteCLIP-based MLLM, trained on new GRES and PreGRES datasets for remote-sensing reasoning segmentation, captioning and VQA.
Problem
Reasoning segmentation from complex implicit queries underperforms on remote-sensing imagery, and geospatial VLMs cannot output segmentation masks.
Approach
Extends the LISA embedding-as-mask paradigm to top-down satellite imagery by pre-training a RemoteCLIP-based Vicuna MLLM on a large geospatial VQA/caption corpus (PreGRES), then fine-tuning with a semi-synthetic GPT-4V/GeoSAM-generated reasoning segmentation dataset (GRES).
Key results
- LISAt achieves gIoU 0.275 and cIoU 0.245 on GRES versus LISA-7B's 0.113 and 0.122.
- LISAt-pre (PreGRES-pretrained) reaches BLEU-4 72.34 and CIDEr 355.32 on UCM-Captions, above RS-GPT4V's 65.74 and 333.23.
- On RSVQA-LR presence accuracy, LISAt-pre reaches 92.36%, the highest among compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the reasoning-segmentation model on LISAt's geospatial setup rather than use LISA-13B | GRES segmentation · GRES All | gIoU: 0.139 ± 0.006 -> 0.275 ± 0.009 | Helped |
| Use LISAt rather than LISA-13B | GRES Small segmentation · GRES Small | gIoU: 0.089 ± 0.007 -> 0.240 ± 0.009 | Helped |
| Use LISAt rather than LISA-13B | GRES Large segmentation · GRES Large | gIoU: 0.244 ± 0.019 -> 0.348 ± 0.015 | Helped |
| Use SAM instead of GeoSAM as segmentation decoder (ablation) | LISAt · GRES All | gIoU: 0.238 ± 0.007 -> 0.275 ± 0.009 | Helped |
| Use RemoteCLIP instead of CLIP as vision encoder (ablation) | LISAt-pre with Vicuna · UCM-Captions | BLEU-4: 66.68 -> 72.34 | Helped |
| Use RemoteCLIP instead of Geo-CLIP as vision encoder (ablation) | LISAt-pre with Vicuna · UCM-Captions | BLEU-4: 24.56 -> 72.34 | Helped |
| Use RemoteCLIP instead of SAT-CLIP as vision encoder (ablation) | LISAt-pre with Vicuna · UCM-Captions | BLEU-4: 16.87 -> 72.34 | Helped |
| Use Vicuna instead of Llama 2 as language model (ablation) | LISAt-pre with RemoteCLIP · UCM-Captions | BLEU-4: 68.31 -> 72.34 | Helped |
| Use RemoteCLIP instead of CLIP336 as vision encoder (ablation) | LISAt-pre with Vicuna · UCM-Captions | BLEU-4: 68.28 -> 72.34 | Helped |
| Use LISAt-pre instead of RS-GPT4V | Remote-sensing captioning · UCM-Captions | BLEU-4: 65.74 -> 72.34 | Helped |
| Use LISAt-pre instead of RS-GPT4V | Remote-sensing captioning · RSICD | BLEU-4: 36.83 -> 36.10 | Hurt |
| Use LISAt-pre instead of RS-GPT4V | Remote-sensing captioning · Sydney-Captions | BLEU-4: 62.23 -> 54.24 | Hurt |
Surprising findings
- GeoSAM masks underperform SAM as the segmentation decoder (cIoU 0.220 vs 0.245), suggesting geospatial pre-training of the segmenter does not help here.
- LISA-7B fine-tuned on GRES performs very poorly on captioning (BLEU-4 8.73 on UCM-Captions), indicating reasoning-segmentation fine-tuning harms text generation.
Limitations
- Struggles with cloudy images, obscured features, and vague queries.
- GRES ground-truth masks from GeoSAM are sometimes incorrect, penalizing correct predictions.
- Performance plateaus as training data grows, suggesting need for more class diversity beyond xView.
Experimental setup
Datasets: PreGRES GRES xView FP-Ref-COCO ReasonSeg NWPU-Captions RSICD RSITMD Sydney-Captions UCM-Captions RSVQA-LR RSVQA-HR
Benchmarks: GRES UCM-Captions NWPU-Captions RSICD Sydney-Captions RSVQA-LR
Metrics: gIoU cIoU BLEU-4 CIDEr SPICE accuracy
Baselines: LISA-7B LISA-13B-Llama2 RS-GPT4V EarthGPT LLaVA-v1.5-7b LLaVA-v1.6-7b GeoChat
Builds on: LISA LLaVA SAM RemoteCLIP
Closest papers by meaning
- VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.90)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.89)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.89)
- TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.88)
- RemoteReasoner: Towards Unifying Geospatial Reasoning Workflow (AAAI 2026 · similarity 0.88)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.