Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts SAM (ViT-B) to shifted downstream segmentation tasks via source-free weakly supervised self-training with an anchor network, contrastive regularization, and LoRA on the encoder, using box/point/polygon prompts as weak supervision.
Problem
SAM generalizes poorly under strong distribution shift (corrupted, camouflaged, medical, robotic images), and adapting a large foundation model is costly and often infeasible without source data or target labels.
Approach
Source-free self-training of SAM's image encoder where a frozen source anchor network regularizes teacher/student predictions, a feature-level contrastive loss regularizes encoder outputs, weak prompts (box, points, coarse polygon) produce cleaner pseudo labels, and only low-rank LoRA weights are updated.
Key results
- On COCO with box prompts, the method reaches 80.12 mIoU vs 74.29 for direct SAM and 77.29 for WDASS, approaching the supervised upper bound of 81.50.
- On camouflaged CHAMELEON with point prompts, the method improves over direct SAM by roughly 35 mIoU points (39.37 to 74.00).
- On COCO-C corruptions with box prompts, average mIoU rises from 66.73 (direct) to 71.05, outperforming all competing adaptation methods on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · COCO-C, box prompt (average) | mIoU: 66.73 -> 71.05 | Helped |
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · COCO 2017, box prompt | mIoU: 74.29 -> 80.12 | Helped |
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · kvasir-SEG, box prompt | mIoU: 81.59 -> 85.47 | Helped |
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · ISIC, box prompt | mIoU: 66.74 -> 80.01 | Helped |
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · CHAMELEON, box prompt | mIoU: 51.32 -> 75.94 | Helped |
| adapt SAM with weakly supervised self-training, anchor regularization, and low-rank finetuning | SAM ViT-B · OSD, box prompt | mIoU: 87.62 -> 92.11 | No clear effect |
| add anchor-loss regularization to self-training (ablation) | weakly supervised SAM adaptation · COCO, box prompt | mIoU: 58.88 -> 79.65 | Helped |
| add contrastive loss to the adaptation objective (ablation) | weakly supervised SAM adaptation · COCO, box prompt | mIoU: 79.65 -> 80.12 | Helped |
| use low-rank encoder adaptation instead of full encoder finetuning (ablation) | SAM ViT-B · COCO, box prompt | mIoU: 78.54 -> 80.12 | Helped |
| finetune MaskDecoder instead of using LoRA (ablation) | SAM ViT-B · COCO, box prompt | mIoU: 80.12 -> 76.81 | Hurt |
| tune explicit visual prompts instead of using LoRA (ablation) | SAM ViT-B · COCO, box prompt | mIoU: 80.12 -> 77.38 | Hurt |
Surprising findings
- Self-training alone causes a large performance drop (COCO box: 74.29 to 58.88) due to confirmation bias, which the anchor loss largely remedies.
- Updating only LoRA in the image encoder outperforms full encoder finetuning (80.12 vs 78.54 IoU) in the ablation.
Limitations
- Relies on ground-truth-derived simulated weak prompts (boxes, 5+5 points, polygons) for training and testing, not real human annotations.
- Uses ViT-B encoder only due to memory constraints; larger SAM variants not evaluated.
- Point-prompt results on some datasets (e.g., COCO without weak supervision) are worse than direct SAM, and the method is not uniformly best (e.g., CAMO poly, OSD point).
Experimental setup
Datasets: COCO Pascal VOC COCO-C Kvasir-SEG ISIC COD10K CHAMELEON CAMO OCID OSD SA-1B
Benchmarks: COCO COCO-C Pascal VOC Kvasir-SEG ISIC CHAMELEON CAMO COD10K OCID OSD
Metrics: mIoU
Baselines: Direct (pre-trained SAM) TENT SHOT Soft Teacher TRIBE DePT WDASS Supervised upper bound
Builds on: Segment Anything (SAM) SHOT LoRA WDASS Soft Teacher
Closest papers by meaning
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.88)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.87)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.87)
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.87)
- BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.