Research map / Detection and segmentation / Open-set and few-shot detection
Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns SAM's prompt and granularity sensitivity into a feature by wrapping SAM in a conditional VAE-style latent model (prompt and image-embedding Gaussians with posterior-distillation KL losses) that produces diverse, plausible segmentations for ambiguous objects.
Problem
SAM gives unstable outputs under small prompt changes and across object granularities, which is a flaw for deterministic segmentation but is also a mismatch for ambiguous segmentation where multiple valid labels exist.
Approach
Model prompt embeddings and image embeddings as latent Gaussians, generated conditionally on the image (PGN/IGN) and trained with a posterior network that sees ground truth and is distilled into the prior network via KL loss; sampling yields multiple candidate masks, with learnable weights ensembling SAM's multi-scale outputs.
Key results
- On LIDC, A-SAM reaches GED 0.228 and HM-IoU 0.717 versus 0.361 and 0.380 for SAM with box shift.
- On LIDC, A-SAM outperforms the best conventional method Mose (GED 0.234, HM-IoU 0.623) with HM-IoU 0.763 and Dmax 0.959 in Table 2.
- On BraTS, A-SAM achieves GED 0.193 and HM-IoU 0.610 versus 0.239 and 0.242 for SAM with box shift; ablation removing prompt ambiguity gives GED 0.308.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use A-SAM instead of the strongest reported baseline | ambiguous segmentation models · LIDC | GED: 0.234 -> 0.230 | Helped |
| use A-SAM instead of the strongest reported baseline | ambiguous segmentation models · BraTS | GED: 0.225 -> 0.192 | Helped |
| use A-SAM instead of the strongest reported baseline | ambiguous segmentation models · ISBI | GED: 0.289 -> 0.267 | Helped |
| use A-SAM instead of the strongest reported baseline | ambiguous segmentation models · SIM 10k | GED: 0.292 -> 0.241 | Helped |
| add prompt and object ambiguity modeling (ablation) | A-SAM · LIDC | GED: 0.361 -> 0.228 | Helped |
| add prompt and object ambiguity modeling (ablation) | A-SAM · LIDC | HM-IOU: 0.380 -> 0.717 | Helped |
| add object ambiguity modeling (ablation) | A-SAM · LIDC | GED: 0.370 -> 0.228 | Helped |
| add object ambiguity modeling (ablation) | A-SAM · LIDC | HM-IOU: 0.389 -> 0.717 | Helped |
| add prompt ambiguity modeling (ablation) | A-SAM · LIDC | GED: 0.308 -> 0.228 | Helped |
| add prompt ambiguity modeling (ablation) | A-SAM · LIDC | HM-IOU: 0.674 -> 0.717 | Helped |
| add posterior distillation (ablation) | A-SAM · LIDC | GED: 0.266 -> 0.228 | Helped |
| add posterior distillation (ablation) | A-SAM · LIDC | HM-IOU: 0.385 -> 0.717 | Helped |
Surprising findings
- Removing the prompt ambiguity module gives higher HM-IoU on LIDC (0.674 vs 0.717 full) but in the ablation table the full model is not uniformly best, e.g. No Posterior Distillation has lower Dmean on some rows.
- SAM's output varies substantially under only 5-pixel box shifts, which the authors frame as a usable source of ambiguity.
Limitations
- Uncertainty is uncontrolled and unquantifiable, so accuracy varies across scenarios.
- Baseline results on BraTS, SIM 10k and other non-quoted datasets are reimplementations from official code, which may not match original settings.
Experimental setup
Datasets: LIDC-IDRI BraTS 2017 ISBI 2016 SIM 10k
Benchmarks: LIDC-IDRI BraTS 2017 ISBI 2016 SIM 10k
Metrics: GED HM-IoU Dmax Dmean
Baselines: SegGPT SEEM SAM Probabilistic U-Net HProb UNet PHiseg SSN CAR PixelSeg CIMD Mose cFlow
Builds on: Segment Anything Model Probabilistic U-Net Mix of Stochastic Experts
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.90)
- UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model (ICML 2025 · similarity 0.88)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.87)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.87)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.86)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.