Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes unsupervised meta-training for few-shot semantic segmentation by clustering self-supervised (MoCo) pixel embeddings into pseudo-classes to build pseudo meta-tasks, plus a Transformer FSS model with prototype-aware cross-attention for robustness to noisy masks.
Problem
Few-shot semantic segmentation models need costly pixel-level annotations on base classes, and small segmentation training sets limit episode diversity and generalization.
Approach
Automatically construct meta-training episodes from unlabeled images by k-means clustering of pre-trained unsupervised pixel embeddings (treating clusters as pseudo-classes), and use a Transformer FSS model whose cross-attention attends to grouped prototypes instead of all support pixels, making training robust to pseudo-mask noise.
Key results
- Fully unsupervised model reaches 54.1% mIoU 1-shot on PASCAL-5i (fold-wise setting outperforms MaskSplit by 6.6% mean mIoU).
- With fine-tuning on only 20% of annotations, 1-shot PASCAL-5i mIoU reaches 64.3%, comparable to CyCTR fully supervised (64.2 vs 63.1 reported in ablation table).
- Prototype-aware cross-attention improves COCO-20i 1-shot mean mIoU from 29.4 to 31.0 over vanilla cross-attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unsupervised rather than supervised meta-training | Ours · PASCAL-5i | mIoU: 63.1 -> 54.1 | Hurt |
| Use unsupervised rather than supervised meta-training | PFENet · PASCAL-5i | mIoU: 60.9 -> 44.0 | Hurt |
| Use unsupervised rather than supervised meta-training | CyCTR · PASCAL-5i | mIoU: 62.9 -> 48.5 | Hurt |
| Add unsupervised meta-training before fine-tuning with 5% labeled data | PFENet · PASCAL-5i | mIoU: 50.5 -> 60.7 | Helped |
| Add unsupervised meta-training before fine-tuning with 20% labeled data | CyCTR · PASCAL-5i | mIoU: 61.3 -> 62.7 | Helped |
| Add unsupervised meta-training before fine-tuning with 20% labeled data | Ours · PASCAL-5i | mIoU: 61.4 -> 64.3 | Helped |
| Replace prototype-aware cross-attention with vanilla cross-attention (ablation) | Ours · COCO-20i | mIoU (%): 31.0 -> 29.4 | Hurt |
| Increase foreground prototypes from 1 to 10, with 10 background prototypes (ablation) | Ours · PASCAL-5i 1-shot | mIoU: 43.0 vs 58.3 | Helped |
| Increase foreground prototypes from 10 to 50, with 100 background prototypes (ablation) | Ours · PASCAL-5i 1-shot | mIoU: 58.2 vs 59.8 | Helped |
| Increase background prototypes from 1 to 100, with 50 foreground prototypes (ablation) | Ours · PASCAL-5i 1-shot | mIoU: 44.5 -> 60.2 | Helped |
Surprising findings
- Unsupervised meta-training alone (0% labels) yields 54.1% PASCAL-5i mIoU, and with 20% labels the model matches or surpasses fully supervised CyCTR and PFENet.
Limitations
- Unsupervised meta-trained model lags supervised training (54.1% vs 63.1% 1-shot on PASCAL-5i) without fine-tuning labels.
- Relies on a self-supervised backbone (MoCo) and k-means cluster count N chosen as a hyperparameter (50 for COCO-20i).
Experimental setup
Datasets: COCO-20i PASCAL-5i ImageNet SDS PASCAL VOC 2012
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU
Baselines: MaskSplit PPNet PFENet RePRI HSNet CyCTR CATrans ASNet BAM
Builds on: MoCo PFENet CyCTR masked average pooling prototypes MaskSplit
Closest papers by meaning
- Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.91)
- A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation (NeurIPS 2024 · similarity 0.90)
- APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Text Augmented Correlation Transformer For Few-shot Classification & Segmentation (CVPR 2025 · similarity 0.90)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.