Research map / Detection and segmentation / Open-set and few-shot detection
Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts ZegCLIP's relationship descriptors to pre-trained ViT patch embeddings, extended across multiple layers, for generalized and binary few-shot semantic segmentation with a frozen backbone, deep prompts, and a lightweight transformer decoder.
Problem
Few-shot semantic segmentation models overfit to base classes seen in training and fail to generalize to novel classes, especially when scaling to multiple support classes in generalized FSS.
Approach
Exploit the patch-level semantic grouping ability of pre-trained ViTs by building per-class relationship descriptors (attention-weighted element-wise products of patch embeddings and momentum-updated class prototypes), computed before any new decoder transformations, and fuse these descriptors from multiple ViT layers as decoder queries.
Key results
- On PASCAL-5i GFSS 1-shot without test-time tuning, ours-multiple reaches 51.0 mIoU(N) and 61.1 hIoU versus 23.2 and 35.2 for CAPL-ViT.
- On PASCAL-5i binary FSS 1-shot, ours-multiple achieves 78.9 mIoU versus 69.3 for DCAMA (Swin-B) and 64.7 for FPTrans (ViT-B/16).
- On COCO-20i GFSS 5-shot, ours-multiple reaches 27.0 mIoU(N) without test-time tuning, versus 11.9 for CAPL-ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-layer relationship descriptors instead of single-layer descriptors (ablation) | Ours · PASCAL-5i GFSS 1-shot, no test-time fine-tuning | mIoU(N): 45.4 -> 51.0 | Helped |
| Add multi-layer relationship descriptors instead of single-layer descriptors (ablation) | Ours · PASCAL-5i GFSS 5-shot, no test-time fine-tuning | mIoU(N): 51.2 -> 55.3 | Helped |
| Add multi-layer relationship descriptors instead of single-layer descriptors (ablation) | Ours · COCO-20i GFSS 1-shot, no test-time fine-tuning | mIoU(N): 17.2 -> 21.1 | Helped |
| Add multi-layer relationship descriptors instead of single-layer descriptors (ablation) | Ours · COCO-20i GFSS 5-shot, no test-time fine-tuning | mIoU(N): 22.0 -> 27.0 | Helped |
| Add relationship descriptors to class-prototype queries (ablation) | Ours · PASCAL-5i GFSS 1-shot | mIoU(N): 15.9 -> 45.4 | Helped |
| Use the proposed patch-based RD instead of an RD built from [CLS] (ablation) | Ours single-layer RD · PASCAL-5i GFSS 1-shot | mIoU(N): 41.9 -> 45.4 | Helped |
| Use the proposed patch-based RDs instead of RDs built from [CLS] (ablation) | Ours multi-layer RDs · PASCAL-5i GFSS 1-shot | mIoU(N): 48.6 -> 51.0 | Helped |
| Use the proposed patch-based RD instead of an RD built from [CLS] (ablation) | Ours single-layer RD · COCO-20i GFSS 5-shot | mIoU(N): 16.8 -> 22.0 | Helped |
| Use the proposed patch-based RDs instead of RDs built from [CLS] (ablation) | Ours multi-layer RDs · COCO-20i GFSS 5-shot | mIoU(N): 20.6 -> 27.0 | Helped |
| Use layers 9-10-11 rather than layer 11 alone for RDs (ablation) | Ours · PASCAL VOC 2012 GFSS 1-shot | mIoU(N): 45.4 -> 51.0 | Helped |
| Use multi-layer relationship descriptors (ablation) | Ours · COCO-20i FSS 5-shot | mIoU: 59.2 -> 61.2 | Helped |
| Use multi-layer relationship descriptors (ablation) | Ours · COCO-20i GFSS 5-shot, test-time fine-tuning | mIoU(N): 25.1 -> 33.1 | Helped |
Surprising findings
- Using the [CLS] token for relationship descriptors (ZegCLIP style) on a plain ViT gives a substantial gain over no RD, but is clearly worse than the proposed patch-based RD.
- Simply swapping the ResNet backbone for ViT-B/16 in CAPL does not fix novel-class generalization (CAPL-ViT remains weak on novel classes), suggesting the RD design, not the backbone, drives gains.
Limitations
- Limited by identification ability and resolution of ViT; similar objects may be misclassified.
- Small objects may be ignored in complex scenes.
- Strong GFSS numbers rely on test-time fine-tuning for some comparisons; base results without it are lower.
Experimental setup
Datasets: PASCAL-5i COCO-20i SDS
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU(N) mIoU(B) hIoU mIoU
Baselines: CAPL CAPL-ViT FineTune CCA DIaM POP PANet PFENet BAM FPTrans PGMA-Net HSNet
Builds on: ZegCLIP CAPL ViT MMSegmentation
Closest papers by meaning
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.91)
- A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation (NeurIPS 2024 · similarity 0.91)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
- Simple Semantic-Aided Few-Shot Learning (CVPR 2024 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.