Research map / Label-efficient and robust learning / Semi-supervised learning
Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Enhances SAM-Med2D for semi-supervised medical image segmentation by generating unsupervised CLIP/VQA/GPT-4 prompts and aligning the decoder with a DPO-inspired loss using simulated annotator ratings of thresholded candidates.
Problem
SAM-based medical segmentation still relies on expert prompts and large annotated datasets, while active learning and reward-model RLHF approaches still require continuous human involvement or explicit reward training.
Approach
Combines unsupervised box and text prompts (BiomedCLIP saliency with CRF, MedVInT VQA answers, GPT-4 generic descriptions) for SAM-Med2D, then fine-tunes the decoder on unlabeled data with a DPO-style loss that rewards higher-rated thresholded candidates and penalizes lower-rated ones, where ratings come from overlap with ground truth as a simulated annotator.
Key results
- On Chest X-ray with 20% of data, the method reaches 78.87 Dice versus 68.41 for Self-prompt and 67.81 for SAM-Med2D.
- On AMOS-CT with 20% of data, mean Dice of 77.69 versus 72.54 for SAM-Med3D and 71.83 for Self-prompt.
- Ranking-based preference scoring outperforms rating and best-candidate-only losses (AMOS-CT mDice 77.69 vs 77.23 and 75.01 at 10% unannotated data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add preference alignment trained with unannotated data (ablation) | SAM-Med2D with unsupervised prompts · Chest-Xray | Dice: 75.60 -> 78.87 | Helped |
| add VQA prompts (ablation) | SAM-Med2D prompting · Chest-Xray | Dice: 73.35 -> 75.60 | Helped |
| add BiomedCLIP visual prompts to GPT-4 textual prompts (ablation) | SAM-Med2D prompting · Chest-Xray | Dice: 57.02 -> 72.76 | Helped |
| use candidate ranking instead of loss for the best candidate only (ablation) | Preference alignment · Chest-Xray | Dice: 77.09 -> 78.87 | Helped |
| use candidate ranking instead of ratings (ablation) | Preference alignment · AMOS-CT | mDice: 77.23 -> 77.69 | Helped |
| increase unannotated training data from 10% to 40% (ablation) | Ranking-based preference alignment · Chest-Xray | Dice: 78.87 -> 89.68 | Helped |
| add VQA and GPT-4 prompts to visual prompts (ablation) | SAM-Med2D prompting · Chest-Xray | Dice: 72.76 -> 73.35 | Helped |
| add the full framework's prompting and preference alignment | SAM-Med2D · Chest X-ray | Dice: 68.41 -> 78.87 | Helped |
Surprising findings
- The 10%+10% unannotated model matches the fully supervised prompt-only model at 20% data on Chest X-ray (78.87 vs 79.13 IoU-Dice comparison in ablation).
Limitations
- Relies on ground truth to simulate annotator ratings, so the semi-supervised setting is simulated rather than tested with real human feedback.
- Requires an initial annotated fine-tuning stage on 10% of the data.
- Depends on GPT-4 and multiple pretrained VLMs, adding external dependencies.
Experimental setup
Datasets: COVID-QU-Ex BUSI UDIAT AMOS-CT
Benchmarks: COVID-QU-Ex BUSI UDIAT AMOS-CT
Metrics: Dice IoU surface Dice similarity mean Dice
Baselines: U-Net nnU-Net SAM SAM-Med2D Self-prompt SAM SAM-Med3D
Builds on: Segment Anything Model SAM-Med2D direct preference optimization BiomedCLIP MedVInT
Closest papers by meaning
- BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation (ICML 2024 · similarity 0.90)
- One-Prompt to Segment All Medical Images (CVPR 2024 · similarity 0.89)
- Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding (CVPR 2024 · similarity 0.89)
- Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model Using 3D Whole-Body CT Scans (AAAI 2025 · similarity 0.88)
- SGTC: Semantic-Guided Triplet Co-training for Sparsely Annotated Semi-Supervised Medical Image Segmentation (AAAI 2025 · similarity 0.88)
- Semi-supervised TEE Segmentation via Interacting with SAM Equipped with Noise-Resilient Prompting (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.