Research map / Detection and segmentation / Open-set and few-shot detection
Towards Open-Vocabulary Remote Sensing Image Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary remote sensing image semantic segmentation (OVRSISS), a 51,846-image LandDiscover50K dataset, and GSNet, which fuses a CLIP generalist stream with a self-supervised DINO-trained RSI specialist stream via query-guided fusion and a residual decoder.
Problem
Existing remote sensing segmentation methods are restricted to a predefined class set and generalize poorly across domains, and no dataset or method exists for segmenting arbitrary classes in remote sensing imagery.
Approach
Dual-stream encoder combining a general CLIP ViT-B/16 with a DINO-pretrained RSI backbone trained on LandDiscover50K images without labels, fused with text queries through query-guided feature fusion and decoded with a residual information preservation decoder.
Key results
- GSNet achieves 31.25 average mIoU across FLAIR, FAST, Potsdam and FloodNet, outperforming the second-best method by 3.54 points.
- On ISPRS Potsdam GSNet reaches 45.75 mIoU versus 38.79 for CAT-SEG trained on LandDiscover50K.
- Replacing CLIP with RemoteCLIP degrades CAT-SEG from 27.64 to 19.88 average mIoU, showing limited generalization of the RSI-specific model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the RSI specialist backbone to CLIP in the dual-stream encoder (ablation) | GSNet · Avg. | mIoU: 27.90 -> 31.25 | Helped |
| Use the RSI specialist backbone alone instead of the dual-stream CLIP and specialist encoder (ablation) | GSNet · Avg. | mIoU: 31.25 -> 6.98 | Hurt |
| Use RemoteCLIP instead of CLIP in the dual-stream encoder (ablation) | GSNet · Avg. | mIoU: 31.25 -> 19.61 | Hurt |
| Add QGFF instead of simple concatenation fusion (ablation) | GSNet · Avg. | mIoU: 18.90 -> 31.25 | Helped |
| Add QGFF instead of Fusion Q (ablation) | GSNet · Avg. | mIoU: 19.81 -> 31.25 | Helped |
| Add the integrated RIPD decoder rather than using the non-integrated decoder (ablation) | GSNet · Avg. | mIoU: +18% | Helped |
| Integrate CLIP features in RIPD (ablation) | GSNet · Avg. | mIoU: +14% | Helped |
| Integrate RSI specialist backbone features in RIPD (ablation) | GSNet · Avg. | mIoU: +23% | Helped |
| Train on the full LandDiscover50K dataset rather than an existing RSISS dataset (ablation) | GSNet · Avg. | mIoU: 15.22 -> 31.25 | Helped |
| Increase the LandDiscover50K training subset size from the smallest subset (ablation) | GSNet · Avg. | mIoU: over 18% | Helped |
| Freeze RSIB instead of fully fine-tuning it, with CLIP attention fine-tuning (ablation) | GSNet · Avg. | mIoU: 27.04 -> 31.25; 4.21 | Helped |
| Freeze RSIB instead of fine-tuning its attention layers, with CLIP attention fine-tuning (ablation) | GSNet · Avg. | mIoU: 28.04 -> 31.25; 3.21 | Helped |
Surprising findings
- Replacing CLIP with the RSI-pretrained RemoteCLIP lowers performance for both CAT-SEG and SCAN, despite the domain-specific pretraining.
- The self-supervised RSI backbone alone (without labels) under-performs strongly, yet it helps when combined with CLIP.
Limitations
- Evaluation datasets are used cross-domain without restricting to novel classes, and the paper notes that Potsdam is in-vocabulary-like, so OVRSISS generalization is only partially isolated.
- GSNet has higher FLOPs (2,160G) than CAT-SEG (660G) and only moderate gains on FLAIR (20.00 vs 22.48 for SAN).
Experimental setup
Datasets: LandDiscover50K Open Earth Map LoveDA DeepGlobe Land Cover SIOR SOTA COCO-Stuff
Benchmarks: FLAIR FAST ISPRS Potsdam FloodNet
Metrics: mIoU GFLOPs parameters
Baselines: CAT-SEG SCAN OV-SEG SAN SED EBSeg RemoteCLIP
Builds on: CAT-SEG CLIP DINO RemoteCLIP
Closest papers by meaning
- SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images (CVPR 2025 · similarity 0.89)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.89)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.88)
- InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition (NeurIPS 2025 · similarity 0.88)
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.88)
- ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.