Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Bayesian entropy decomposition of SAM uncertainty into model, prompt, and task components, and USAM, lightweight MLPs that estimate these uncertainties deterministically from SAM tokens.
Problem
Uncertainty quantification for the Segment Anything Model is hard because SAM is class- and task-agnostic: uncertainty comes from under-parameterized models, insufficient prompts, and ambiguous tasks, and existing UQ methods (MC sampling, ensembles, TTA, SamScore) are costly or do not separate these
Approach
Decompose SAM predictive uncertainty into epistemic (model size), aleatoric prompt, and task components via a Bayesian entropy approximation, then train post-hoc MLPs on SAM's mask and IoU tokens to directly predict expected IoU and the gaps between uncertainty sources, avoiding sampling at test time.
Key results
- For segmentation uncertainty on COCO with the Tiny model, USAM_L reaches 90.01% rel. AUC, above the best Bayesian baseline H_Theta (78.26%) and SamScore (79.60%) in the main correction table.
- For task uncertainty, the USAM gap Δ_A reaches about 93-95% rel. AUC across datasets, while Bayesian H_A reaches 52-66% on most datasets.
- USAM adds almost no runtime over plain SAM (e.g., 0.441 vs 0.437 s/iteration for Large), while 5 image augmentations raise it to 2.187 s/iteration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use USAM_L to estimate segmentation uncertainty instead of SamScore | SAM Large · DAVIS, ADE20k, MOSE, COCO, and SA-V | rel. AUC (%): 91.59; 92.60; 92.78; 90.01; 89.37 | Helped |
| use direct USAM model-gap estimate Delta*_Theta instead of SamScore | SAM Tiny-to-Large model selection · ADE20k and COCO | rel. AUC (%): 52.85 -> 61.55; 57.97 -> 66.60 | Helped |
| use Bayesian model entropy H_Theta instead of SamScore | SAM Tiny-to-Large model selection · DAVIS, MOSE, and SA-V | rel. AUC (%): 64.25 -> 73.46; 68.44 -> 75.60; 58.83 -> 65.66 | Helped |
| use direct USAM prompt-gap estimate Delta*_XP instead of SamScore | SAM prompt refinement · DAVIS, ADE20k, MOSE, COCO, and SA-V | rel. AUC (%): 75.53; 83.41; 74.50; 78.89; 71.27 | Helped |
| use USAM task-gap estimate Delta_A instead of Bayesian task entropy H_A | SAM task uncertainty · DAVIS, ADE20k, MOSE, COCO, and SA-V | rel. AUC (%): 94.05; 93.08; 94.21; 94.85; 94.17 | Helped |
| use full predictive entropy H_Y instead of SamScore | SAM overall uncertainty · DAVIS, ADE20k, MOSE, COCO, and SA-V | rel. AUC (%): 83.36 -> 38.79; 75.40 -> 46.87; 89.48 -> 56.58; 79.60 -> 45.36; 80.80 -> 57.32 | Hurt |
| add the mask token to the MLP input (ablation) | USAM · COCO | model uncertainty rel. AUC (%): 61.19% -> 63.66% | Helped |
| add the IoU token to the MLP input (ablation) | USAM · COCO | model uncertainty rel. AUC (%): 62.63% -> 63.66% | Helped |
| add the mask token to the MLP input (ablation) | USAM · COCO | prompt uncertainty rel. AUC (%): 72.08% -> 78.30% | Helped |
| add the IoU token to the MLP input (ablation) | USAM · COCO | prompt uncertainty rel. AUC (%): 76.42% -> 78.30% | Helped |
| add the mask token to the MLP input (ablation) | USAM · COCO | task uncertainty rel. AUC (%): 86.89% -> 94.82% | Helped |
| add the IoU token to the MLP input (ablation) | USAM · COCO | task uncertainty rel. AUC (%): 91.96% -> 94.82% | Helped |
Surprising findings
- The Bayesian predictive entropy H_Y performs poorly on the practical tasks, below even mask-entropy baselines, for model and task selection.
- SamScore is only weakly correlated with the true IoU (0.49 on COCO), lower than the post-hoc USAM IoU estimator (0.71).
Limitations
- The Bayesian approximation relies on heuristic probability choices (uniform prompt and model priors, SamScore-normalized task probabilities).
- USAM is weak on the small DAVIS dataset for model uncertainty, suggesting it needs more data variance.
- All methods are applied post-hoc; integration into SAM training is not explored.
Experimental setup
Datasets: SA-V MOSE ADE20k DAVIS COCO
Benchmarks: DAVIS ADE20k MOSE COCO SA-V
Metrics: relative AUC of mIoU under uncertainty-ordered correction Pearson correlation runtime (seconds per iteration)
Baselines: SamScore mask entropy H_Std Bayesian predictive entropy H_Y Bayesian epistemic entropy H_Theta Bayesian prompt entropy H_XP Bayesian task entropy H_A
Builds on: Segment Anything Model deep ensembles test-time augmentation SMAC3
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.91)
- ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation (ICLR 2024 · similarity 0.91)
- Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM (NeurIPS 2024 · similarity 0.88)
- Predictive Uncertainty Quantification for Bird's Eye View Segmentation: A Benchmark and Novel Loss Function (ICLR 2025 · similarity 0.88)
- PTQ4SAM: Post-Training Quantization for Segment Anything (CVPR 2024 · similarity 0.87)
- UNICL-SAM: Uncertainty-Driven In-Context Segmentation with Part Prototype Discovery (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.