Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Enhancing Generalized Few-Shot Semantic Segmentation via Effective Knowledge Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GFSS-EKT, a two-phase generalized few-shot segmentation method that transfers base-class knowledge to novel classes via prototype modulation, classifier weight calibration, and context-consistency learning on unlabeled base samples.
Problem
Generalized few-shot semantic segmentation must segment both base and novel classes; two-phase training leaves a distribution gap between base and novel classes, and novel classes lack contextual information from few samples.
Approach
Transfers knowledge from base to novel classes through three modules: cross-attention modulation of novel prototypes from base prototypes, mean/std calibration of the novel classifier toward the base classifier's weight distribution, and a consistency loss on base images with a context-oriented cutout placed near object bounding-box boundaries.
Key results
- On PASCAL-5i 1-shot, H-Mean 55.47 vs. 47.97 for prior best POP (BCM 52.22); novel mIoU 43.93.
- On COCO-20i 1-shot, H-Mean 31.22 vs. 26.69 for BCM (prior best), a gain of about 4.5 points.
- Ablation on PASCAL-5i 1-shot: removing NCC drops base mIoU by 3.89 points; removing CCL drops novel mIoU by 3.34 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GFSS-EKT instead of POP | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 47.97 -> 55.47 | Helped |
| Use GFSS-EKT instead of POP | GFSS-EKT · COCO-20^i 1-shot | H-Mean: 23.92 -> 31.22 | Helped |
| Add auxiliary loss to base-class pre-training (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 48.58 -> 50.29 | Helped |
| Add NCC to auxiliary loss (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 50.29 -> 51.51 | Helped |
| Add CCL to auxiliary loss and NCC (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 51.51 -> 54.82 | Helped |
| Add NPM to auxiliary loss, NCC, and CCL (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 54.82 -> 55.47 | Helped |
| Use cross-attention instead of cosine similarity for prototype reconstruction (ablation) | NPM · PASCAL-5^i 1-shot | Novel mIoU: 41.32 -> 43.93 | Helped |
| Calibrate the novel classifier only, rather than removing calibration (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 50.29 -> 51.51 | Helped |
| Calibrate both base and novel classifiers instead of calibrating the novel classifier only (ablation) | GFSS-EKT · PASCAL-5^i 1-shot | H-Mean: 51.51 -> 50.18 | Hurt |
| Use unlabeled rather than labeled base samples during fine-tuning (ablation) | CCL · PASCAL-5^i 1-shot | Novel mIoU: 39.05 -> 42.49 | Helped |
| Use both classifiers rather than omitting either classifier from consistency prediction (ablation) | CCL · PASCAL-5^i 1-shot | H-Mean: 49.20 -> 54.82; 48.01 -> 54.82 | Helped |
| Use only base samples rather than only novel or both sets for consistency learning (ablation) | CCL · PASCAL-5^i 1-shot | H-Mean: 51.65 -> 54.82; 53.31 -> 54.82 | Helped |
Surprising findings
- Using base samples with labels in fine-tuning hurts performance versus the label-free consistency approach (H-Mean 51.27 vs 54.82 in 1-shot).
- Calibrating both base and novel classifiers (NBCC) performed worse than calibrating only the novel classifier on base-class accuracy.
Limitations
- Performance gains over prior methods diminish in the 5-shot setting, as base-class complementary knowledge shrinks with more novel samples.
- Novel-class performance remains low in absolute terms (e.g., 21.83 novel mIoU on COCO-20i 1-shot).
Experimental setup
Datasets: PASCAL-5i COCO-20i
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU Base mIoU Novel mIoU Mean H-Mean
Baselines: CAPL PKL DIaM POP BCM
Builds on: POP PSPNet CAPL PKL DIaM
Closest papers by meaning
- A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation (NeurIPS 2024 · similarity 0.94)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.87)
- Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.87)
- Simple Semantic-Aided Few-Shot Learning (CVPR 2024 · similarity 0.87)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.