Research map / Detection and segmentation / Open-set and few-shot detection
SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SegEarth-OV, a training-free open-vocabulary segmentation method for remote sensing images that combines a lightly trained feature upsampler (SimFeatUp) with subtraction of the [CLS] global bias from CLIP patch tokens.
Problem
Open-vocabulary semantic segmentation of remote sensing images, where natural-image CLIP-based OVSS methods give distorted shapes and ill-fitting boundaries due to low-resolution features and global bias in patch tokens.
Approach
A FeatUp-derived upsampler (SimFeatUp) trained on unlabeled remote sensing images with an added image reconstruction loss, applied to earlier-layer CLIP features, plus subtracting the scaled [CLS] token from patch tokens to reduce global bias; no labels or segmentation training are needed.
Key results
- SegEarth-OV reaches 39.2 average mIoU over 8 remote sensing semantic segmentation datasets, versus 33.4 for the best prior training-free method (ClearCLIP), a 5.8% average gain.
- On flood detection (WBS-SI), SegEarth-OV achieves 60.2 IoU, a 15.3% improvement over ClearCLIP's 44.9.
- Plugging SimFeatUp and global bias alleviation into ClearCLIP raises OpenEarthMap mIoU from 31.0 to 39.1 and WHUAerial IoU from 36.6 to 51.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Upsample earlier-stage CLIP features (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 32.4 -> 34.6 | Helped |
| Use remote-sensing Million-AID images to train the upsampler (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 34.6 -> 36.0 | Helped |
| Replace the JBU stack with one JBU module repeated as needed (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 36.0 -> 36.3 | Helped |
| Add image reconstruction loss (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 36.3 -> 37.6 | Helped |
| Subtract the [CLS] token to alleviate global bias (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 37.6 -> 39.3 | Helped |
| Expand the upsampling kernel to 11 × 11 (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 39.3 -> 40.3 | Helped |
| Use the original FeatUp upsampler (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 32.4 -> 33.9 | Helped |
| Use the original FeatUp upsampler (ablation) | SegEarth-OV · WHUSat.II | IoU: 22.7 -> 20.2 | Hurt |
| Use FeatUp with the MaskCLIP value features (ablation) | SegEarth-OV · OpenEarthMap | mIoU: 32.4 -> 33.8 | Helped |
| Use FeatUp with the MaskCLIP value features (ablation) | SegEarth-OV · WHUSat.II | IoU: 22.7 -> 25.2 | Helped |
| Use the original FeatUp upsampler (ablation) | SegEarth-OV · WBS-SI | IoU: 46.9 -> 39.6 | Hurt |
Surprising findings
- Original FeatUp upsampling decreases performance on WHUSat.II and WBS-SI, so generic upsamplers do not directly help training-free OVSS.
- Remote sensing CLIP models (RemoteCLIP, GeoRSCLIP, SkyCLIP) do not outperform vanilla CLIP on OVSS in the reported comparison, with RemoteCLIP performing worse than CLIP.
Limitations
- Road extraction performance remains low (best IoU 35.4%) and the authors attribute this to road shape and imprecise OpenStreetMap-derived labels.
- Performance on fine-grained datasets such as iSAID is low (21.7 mIoU), attributed to 16 fine-grained categories.
- SimFeatUp still requires training on a few unlabeled images, so the method is only approximately training-free.
Experimental setup
Datasets: Million-AID OpenEarthMap LoveDA iSAID Potsdam Vaihingen UAVid UDD5 VDD WHUAerial WHUSat.II Inria
Benchmarks: OpenEarthMap LoveDA iSAID Potsdam Vaihingen UAVid UDD5 VDD WHUAerial WHUSat.II Inria xBD
Metrics: mIoU IoU
Baselines: CLIP MaskCLIP SCLIP GEM ClearCLIP FeatUp TCL Reco
Builds on: FeatUp CLIP MaskCLIP ClearCLIP JBU
Closest papers by meaning
- Towards Open-Vocabulary Remote Sensing Image Semantic Segmentation (AAAI 2025 · similarity 0.89)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.86)
- Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.85)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.85)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.