Research map / Detection and segmentation / Open-set and few-shot detection
Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-scale visual-prompted transformer decoder with novel-to-base causal attention for generalized few-shot segmentation, with optional transductive prompt tuning.
Problem
Generalized few-shot semantic segmentation requires strong accuracy on novel classes from few examples while retaining base-class performance; prompting of transformer decoders for dense prediction is under-explored.
Approach
Learnable per-class visual prompts initialized by masked average pooling of support features, refined across decoder scales, with a shared unidirectional novel-to-base causal cross-attention that lets novel prompts be contextualized by frozen base prompts without altering base prompts.
Key results
- On COCO-20i 1-shot, inductive method reaches 34.78 mean mIoU vs 29.00 for inductive DIaM; transductive version reaches 36.05 vs 32.75 for transductive DIaM.
- On PASCAL-5i 5-shot, transductive method reaches 66.27 mean mIoU vs 63.08 for transductive DIaM.
- Ablation on COCO-20i shows novel-to-base causal attention raises 1-shot novel mIoU from 11.31 to 18.00 with base mIoU essentially preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use transductive prompt tuning | Ours · COCO-20i 1-shot and 5-shot | mean mIoU: 34.78 -> 36.05; 40.83 -> 42.48 | Helped |
| Use transductive prompt tuning | Ours · PASCAL-5i 1-shot and 5-shot | mean mIoU: 54.79 -> 58.11; 62.60 -> 66.27 | Helped |
| Add masked-pooling prompt initialization instead of random initialization, without causal attention (ablation) | Visual prompting method · COCO-20i 1-shot | mean mIoU: 30.70 -> 30.87 | Helped |
| Add novel-to-base causal attention at the first layer (ablation) | Visual prompting method with masked-pooling prompts · COCO-20i 1-shot | mean mIoU: 30.87 -> 31.90 | Helped |
| Apply novel-to-base causal attention at every decoder layer rather than only the first layer (ablation) | Visual prompting method · COCO-20i 1-shot | mean mIoU: 31.90 -> 34.78 | Helped |
| Share causal-attention weights across layers rather than use separate weights (ablation) | Visual prompting method · COCO-20i 1-shot | mean mIoU: 34.56 -> 34.78 | Helped |
| Add masked-pooling prompt initialization instead of random initialization, with shared causal-attention weights (ablation) | Visual prompting method · COCO-20i 1-shot | mean mIoU: 34.08 -> 34.78 | Helped |
| Add transductive prompt tuning (ablation) | Visual prompting method with shared causal-attention weights and masked-pooling prompts · COCO-20i 1-shot | mean mIoU: 34.78 -> 36.05 | Helped |
| Use separate causal-attention weights across decoder layers (ablation) | Visual prompting method · COCO-20i 1-shot | base mIoU: 51.55 -> 51.06 | Hurt |
Surprising findings
- The inductive-only model outperforms transductive DIaM in most COCO-20i and PASCAL-5i settings despite faster inference.
- Separate per-layer causal attention weights give little novel gain and reduce base accuracy, so weight sharing across scales is preferred.
Limitations
- Transductive variant requires per-image test-time optimization, increasing inference time.
- Novel-class performance remains well below base-class performance (e.g., about 18% novel vs 52% base on COCO-20i 1-shot).
Experimental setup
Datasets: COCO-20i PASCAL-5i
Benchmarks: COCO-20i PASCAL-5i
Metrics: mIoU base mIoU novel mIoU harmonic-style mean of base and novel mIoU
Baselines: CAPL BAM DIaM POP RePRI PANET CANeT PFENET SCL MiB
Builds on: Mask2Former DIaM visual prompt tuning BAM
Closest papers by meaning
- Visual In-Context Prompting (CVPR 2024 · similarity 0.89)
- SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation (AAAI 2025 · similarity 0.89)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.89)
- Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.88)
- Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.88)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.