Research map / Label-efficient and robust learning / Recognition and long-tail
Uncertainty-aware Fine-tuning of Segmentation Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUM, a SAM fine-tuning framework that combines a small human-annotated set with SAM-pseudo-labeled unlabeled images, using a learned per-pixel uncertainty map to weight the loss and guide prompt sampling plus a task prompt to disambiguate segmentation type.
Problem
Lightweight fine-tuning of SAM on high-quality annotated data improves accuracy on intricate structures but causes forgetting of SAM's generalization to segmentation tasks (entity, part) absent from the fine-tuning set, while SAM pseudo-labels on unlabeled data are noisy, especially at boundaries.
Approach
Estimate pseudo-label uncertainty via the difference between SAM logits and a refined mask from a separately fine-tuned mask-refinement decoder; use this uncertainty to down-weight focal and Dice losses on unlabeled data, to sample interactive prompts from certain regions, and add a learned task prompt embedding indicating salient-object, entity, or unlabeled data source.
Key results
- SUM achieves 3-point-prompt overall mIoU of 81.0 versus 77.2 for SAM and 77.7 for Vanilla fine-tuning on FT-Small (Table 1).
- SUM outperforms SAM by about 3-6 mIoU and 4-7 mBIoU across point-prompt rounds, according to the abstract.
- On the single-point ablation with FT-Medium, SUM reaches 73.4 overall mIoU versus 69.0 for SAM and 69.3 for Vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with uncertainty-aware losses and prompts using labeled data and SAM pseudo-labels | SUM · Overall average, 3-point prompts | mIoU: 80.3 -> 82.1 | Helped |
| add task prompt (ablation) | SUM · Overall average, 1-point prompt | mIoU: 71.0 -> 73.4 | Helped |
| add task prompt (ablation) | SUM · Overall average, 3-point prompts | mIoU: 82.0 -> 82.1 | No clear effect |
| use task prompt continuously in all interactive rounds instead of only the first (ablation) | SUM · Overall average, 3-point prompts | mIoU: 82.1 -> 81.0 | Hurt |
| use uncertainty-aware training instead of direct mask refinement (ablation) | SUM · Overall average, 1-point prompt | mIoU: 70.3 -> 73.4 | Helped |
| use mask refinement rather than vanilla pseudo-label training (ablation) | Vanilla · Overall average, 1-point prompt | mIoU: 69.3 -> 70.3 | Helped |
| fine-tune with uncertainty-aware pseudo-label use and task prompts | SUM · Part, 3-point prompts | mIoU: 64.4 -> 67.9 | Helped |
| use SUM uncertainty quantification instead of confidence-based uncertainty | SUM · Overall average, 3-point prompts | mIoU: 77.0 -> 82.1 | Helped |
| use SUM uncertainty quantification instead of discrepancy-based uncertainty | SUM · Overall average, 3-point prompts | mIoU: 78.4 -> 82.1 | Helped |
| fine-tune only on human-annotated salient-object data | HQ-SAM and other lightweight fine-tuning strategies · Entity and part segmentation | mIoU | Hurt |
Surprising findings
- Fine-tuning SAM on high-quality salient-object data (HQ-SAM, lightweight adapters, LoRA) degrades entity and part segmentation below SAM, i.e., models forget how to segment anything.
- Confidence-based and discrepancy-based uncertainty estimates, as used in U2PL and prior work, underperform the proposed refinement-based uncertainty.
Limitations
- Quality of the mask-refinement module depends on the initial performance of the SAM model.
- Naive replacement of pseudo-labels with refined masks is less effective than uncertainty-aware training.
- Part-segmentation gains are limited relative to the annotated tasks; semi-supervised baselines are adapted rather than tuned to the setting.
Experimental setup
Datasets: SA-250K HQSeg-44K EntitySeg Internal Dataset COIFT DIS ThinObject5K HR-SOD VOCEVAL BIG Fashionpedia Multi-Human Parsing
Benchmarks: COIFT DIS validation ThinObject5K validation HR-SOD VOCEVAL BIG EntitySeg validation Fashionpedia Multi-Human Parsing Easyportrait Paco
Metrics: mIoU mBIoU
Baselines: SAM HQ-SAM Vanilla fine-tuning AugSeg U2PL Adapter FT LoRA FT
Builds on: Segment Anything Model HQ-SAM U2PL AugSeg
Closest papers by meaning
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.92)
- UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model (ICML 2025 · similarity 0.91)
- Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM (NeurIPS 2024 · similarity 0.90)
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.89)
- InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective (ICML 2025 · similarity 0.89)
- UNICL-SAM: Uncertainty-Driven In-Context Segmentation with Part Prototype Discovery (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.