Research map / Label-efficient and robust learning / Recognition and long-tail
Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H-SAM, a prompt-free, LoRA-adapted SAM with a two-stage hierarchical mask decoder (class-balanced mask-guided self-attention, learnable mask cross-attention, hierarchical pixel decoder) for data-efficient medical image segmentation.
Problem
Adapting SAM to medical image segmentation without expensive full fine-tuning or click/box prompts, especially with few labeled samples and class imbalance.
Approach
A two-stage hierarchical decoder where SAM's original mask decoder produces a prior probabilistic mask that guides a second Transformer decoder via class-balanced Gaussian-noise-augmented mask-guided self-attention and a learnable (untransformed probabilistic) mask cross-attention, plus a U-shaped hierarchical pixel decoder with skip connections.
Key results
- On Synapse with 10% training slices, H-SAM reaches 80.35% mean Dice vs. 75.57% for SAMed, outperforming prompt-free SAM variants by about 4.78%.
- On LA with 4 labeled scans, H-SAM achieves 89.22% Dice vs. 88.02% for BCP (semi-supervised) and 87.72% for SAMed, without unlabeled data.
- On PROMISE12 with 3 labeled cases, H-SAM achieves 87.27% Dice vs. 78.27% for MLB-Seg (semi-supervised) and 86.00% for SAMed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use H-SAM instead of SAMed | SAM adaptation · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 80.35 | Helped |
| Use H-SAM instead of MERIT | medical image segmentation · Synapse multi-organ CT, fully supervised | Mean Dice: 84.90 -> 86.49 | Helped |
| Use H-SAM instead of SAMed | SAM adaptation · LA, 4 labeled scans | Mean Dice: 87.72% vs. 89.22% | Helped |
| Use H-SAM instead of MLB-Seg | few-shot prostate segmentation · PROMISE12, 3 labeled cases | Mean Dice: 78.27 -> 87.27 | Helped |
| Add learnable mask cross-attention (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 77.68; 2.1% | Helped |
| Replace learnable mask-attention with original mask-attention (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 77.68 -> 75.61 | Hurt |
| Add class-balanced mask-guided self-attention (CMAttn) (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 77.05; 1.2% | Helped |
| Add hierarchical pixel decoder (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 77.71; 2.2% | Helped |
| Add learnable mask cross-attention and CMAttn (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 78.58; 3.2% | Helped |
| Add learnable mask cross-attention and hierarchical pixel decoder (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 79.03; 3.0% | Helped |
| Add CMAttn and hierarchical pixel decoder (ablation) | H-SAM decoder · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 78.76; 3.5% | Helped |
| Increase SAMed decoder from 2 to 4 transformer layers (ablation) | SAMed · Synapse multi-organ CT, 10% | Mean Dice: 75.57 -> 76.80 | Helped |
Surprising findings
- Without any unlabeled data, H-SAM with few labeled cases outperforms semi-supervised methods trained with extensive unlabeled data.
- Unlearnable original mask attention gave almost no improvement (75.61% vs 75.57%), while the learnable variant gave 77.68%.
Limitations
- Evaluated only on three medical datasets with multi-organ CT, LA, and prostate MR.
- Relies on a two-stage decoder and extra mask-guided modules, adding architectural complexity over SAMed.
- Fully-supervised Synapse comparison uses 224x224 resolution, which may differ from the few-shot 512x512 setting.
Experimental setup
Datasets: Synapse multi-organ CT LA (2018 Atrial Segmentation Challenge) PROMISE12
Benchmarks: Synapse multi-organ CT LA PROMISE12
Metrics: Dice Hausdorff distance
Baselines: AutoSAM SAM Adapter SAMed TransUnet SwinUnet TransDeepLab DAE-Former MERIT UA-MT SS-Net BCP MLB-Seg
Builds on: Segment Anything Model SAMed LoRA Mask2Former logit adjustment
Closest papers by meaning
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.89)
- Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model Using 3D Whole-Body CT Scans (AAAI 2025 · similarity 0.88)
- EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation (CVPR 2024 · similarity 0.88)
- BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation (ICML 2024 · similarity 0.87)
- A Semantic Knowledge Complementarity based Decoupling Framework for Semi-supervised Class-imbalanced Medical Image Segmentation (CVPR 2025 · similarity 0.87)
- Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.