Research map / Label-efficient and robust learning / Recognition and long-tail
UNICL-SAM: Uncertainty-Driven In-Context Segmentation with Part Prototype Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNICL-SAM, an in-context segmentation generalist that models reference-example uncertainty with a Gaussian graph network, uses uncertainty-guided graph augmentation and feature refinement, and adds part prototypes as visual instructions.
Problem
In-context segmentation generalists are sensitive to the quality and variability of support examples (noisy or mislabeled, degraded), and prior work only filters examples at test time without robust training.
Approach
An uncertainty Gaussian graph network (UGGN) estimates per-patch data uncertainty from reference examples; this uncertainty guides graph augmentation (KL consistency), feature reweighting via a learnable gate, and an uncertainty-weighted graph-clustering stage that yields part prototypes used as extra in-context instructions alongside global pooled features.
Key results
- Achieves 77.8 one-shot and 78.7 few-shot MIoU on COCO-20i, outperforming prior generalists.
- Reaches 34.1 one-shot MIoU on LVIS-92i, surpassing previous generalists.
- Average decline of only 3.1 MIoU under degraded support examples in the robustness simulation test, versus 5.2-8.4% decline patterns for baselines (3.9% average decline ratio).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add part prototype discovery (ablation) | UNICL-SAM without the proposed components · COCO-20i | MIoU: 71.77 -> 72.99 | Helped |
| Add uncertainty probabilistic modeling (ablation) | UNICL-SAM without the proposed components · COCO-20i | MIoU: 71.77 -> 72.41 | Helped |
| Add multi-scale feature adapter, part prototype discovery, and uncertainty probabilistic modeling (ablation) | UNICL-SAM · COCO-20i | MIoU: 71.77 -> 74.64 | Helped |
| Add uncertainty-guided graph augmentation and feature refinement (ablation) | UNICL-SAM · COCO-20i | MIoU: 73.78 -> 74.64 | Helped |
| Replace deterministic GCN with UGGN (ablation) | UNICL-SAM without uncertainty-guided graph augmentation · COCO-20i | MIoU: 72.12 -> 72.41 | Helped |
| Add uncertainty-guided graph augmentation (ablation) | UNICL-SAM with UGGN · COCO-20i | MIoU: 72.41 -> 73.82 | Helped |
| Replace learnable gate with top ratio masking for feature refinement (ablation) | UNICL-SAM · COCO-20i | MIoU: 74.64 -> 43.73 | Hurt |
| Replace learnable gate with random masking for feature refinement (ablation) | UNICL-SAM · COCO-20i | MIoU: 74.64 -> 73.44 | Hurt |
| Use 10 part prototype clusters instead of 5 (ablation) | UNICL-SAM · COCO-20i | MIoU: 73.36 -> 73.82 | Helped |
| Use 15 part prototype clusters instead of 10 (ablation) | UNICL-SAM · COCO-20i | MIoU: 73.82 -> 72.50 | Hurt |
| Use 20 part prototype clusters instead of 10 (ablation) | UNICL-SAM · COCO-20i | MIoU: 73.82 -> 72.38 | Hurt |
| Use UNICL-SAM instead of SINE | COCO-20i one-shot · COCO-20i | MIoU: 64.5 -> 77.8 | Helped |
Limitations
- Robustness test set is self-constructed with simulated degradations rather than real-world data.
- Ablations use a reduced COCO subset (34k images) rather than full training.
Experimental setup
Datasets: COCO ADE20k FSS-1000 LVIS-92i
Benchmarks: COCO-20i FSS-1000 LVIS-92i constructed robustness test set
Metrics: MIoU MAE
Baselines: HSNet VAT FPTrans Painter SegGPT PerSAM Matcher VRP-SAM ICVU SINE
Builds on: SegGPT SEGIC SINE Matcher SAM
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.89)
- ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation (ICLR 2024 · similarity 0.87)
- UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model (ICML 2025 · similarity 0.87)
- A Simple Image Segmentation Framework via In-Context Examples (NeurIPS 2024 · similarity 0.86)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.