Research map / Detection and segmentation / Open-set and few-shot detection
InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructCDS (instruction-oriented counting, detection, segmentation) with the EarthInstruct benchmark and InstructSAM, a training-free pipeline combining an LVLM counter, SAM2 mask proposals, and counting-constrained binary integer programming mask-label matching with GeoRSCLIP.
Problem
Language-guided object recognition in remote sensing imagery relies on explicit category cues and fixed thresholds, so it struggles with complex, implicit, open-ended, or open-subclass instructions and suffers from scarce semantically rich labeled data.
Approach
Decompose instruction-driven recognition into LVLM-based category and count prediction, class-agnostic SAM2 mask proposals, and a confidence-free mask-label assignment solved as a binary integer program that maximizes CLIP similarity subject to the predicted per-category counts.
Key results
- On EarthInstruct open-vocabulary zero-shot setting, InstructSAM-GPT4o reaches 83.0 counting F1 and 41.8 box F1 on NWPU-VHR-10 versus 68.0 and 36.4 for Qwen2.5-VL.
- In the open-ended setting, InstructSAM reduces output tokens by 89% and total inference time by 32% compared to Qwen2.5-VL, with near-constant runtime across object counts.
- InstructSAM-GPT4o achieves 47.9 counting F1 on DIOR open-ended, versus 39.2 for SkySenseGPT and 36.6 for Qwen2.5-VL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InstructSAM-GPT4o instead of Qwen2.5-VL | Open-vocabulary object recognition · NWPU-VHR-10 | Cnt F1: 68.0 -> 83.0 | Helped |
| Use InstructSAM-GPT4o instead of Qwen2.5-VL | Open-vocabulary object recognition · DIOR | Cnt F1: 52.0 -> 79.9 | Helped |
| Use InstructSAM-GPT4o instead of Qwen2.5-VL | Open-ended object recognition · NWPU-VHR-10 | Cnt F1: 48.6 -> 57.4 | Helped |
| Use InstructSAM-GPT4o instead of Qwen2.5-VL | Open-ended object recognition · DIOR | Cnt F1: 36.6 -> 47.9 | Helped |
| Add dataset-specific annotation instructions (ablation) | Qwen2.5-VL counting · NWPU-VHR-10 vehicle counting | F1: 19 -> 41 | Helped |
| Add dataset-specific annotation instructions (ablation) | GPT-4o counting · NWPU-VHR-10 vehicle counting | F1: 20 -> 71 | Helped |
| Add dataset-specific annotation instructions (ablation) | Qwen2.5-VL counting · DIOR | mF1: +3% | Helped |
| Add dataset-specific annotation instructions (ablation) | GPT-4o counting · DIOR | mF1: +8% | Helped |
| Use remote-sensing CLIP instead of generic DFN2B-L (ablation) | InstructSAM with GPT-4o and SAM2-L · NWPU-VHR-10 OVD | Box F1: 41.3 -> 45.8 | Helped |
| Use a smaller SAM2-S and SkyCLIP-B with Qwen2.5-VL (ablation) | InstructSAM · NWPU-VHR-10 OVD | Box F1: 36.4 -> 40.6 | Helped |
| Use counting-constrained mask-label matching instead of fixed CLIP-score thresholding (ablation) | InstructSAM · Multi-class and open-world object recognition | F1 | Helped |
| Use SAM2-S instead of SAM2-L (ablation) | InstructSAM with SkyCLIP-B · NWPU-VHR-10 OVD | Box F1: 45.2 -> 44.1 | Hurt |
Surprising findings
- Explicit dataset-specific annotation rules in prompts substantially improve counting precision, suggesting capable LVLMs can match closed-set detectors with good instructions.
- Generic detectors such as OWLv2 perform strongly for means of transport but poorly for sports fields, reflecting natural-image vocabulary bias.
Limitations
- Performance inherits biases of foundation models trained mostly on optical imagery; GPT-4o and SAM struggle with SAR images.
- SAM2 can miss the full extent of objects with complex geometry (e.g., basketball courts), and scene-level CLIP models align poorly with object-level crops.
- Evaluation covers only two datasets (NWPU-VHR-10 and DIOR) with 20 categories, and open-vocabulary results are reported for a limited set of models.
Experimental setup
Datasets: NWPU-VHR-10 DIOR SAMRS
Benchmarks: EarthInstruct
Metrics: precision recall F1-score mF1 mAP_nc mIoU box F1 mask F1
Baselines: Grounding DINO OWLv2 Qwen2.5-VL GSNet SegEarth-OV GPT-4o + OWL SkySenseGPT EarthDial LAE-Label GeoPixel Faster-RCNN
Builds on: SAM2 CLIP Grounding DINO Qwen2.5-VL VL-SAM
Closest papers by meaning
- Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community (AAAI 2025 · similarity 0.92)
- See What We Cannot See: A Geo-guided Reasoning Benchmark for Object Counting under Adverse Earth Observation Conditions (CVPR 2026 · similarity 0.90)
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.89)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.89)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.88)
- Towards Open-Vocabulary Remote Sensing Image Semantic Segmentation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.