Research map / Detection and segmentation / Open-set and few-shot detection
BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BLO-SAM, which finetunes SAM's mask decoder (LoRA plus transposed convolutions and MLP head) with a learnable prompt embedding under bi-level optimization on two disjoint halves of a tiny training set, enabling prompt-free semantic segmentation with less overfitting.
Problem
SAM needs manual prompts for segmentation and finetuning it on very small labeled datasets (especially medical) overfits, as it updates weights and prompt-like embeddings on the same data.
Approach
Treat the learnable prompt embedding as a hyper-parameter: optimize SAM's weights on one data split (lower level) and the prompt embedding on a disjoint validation split (upper level), using one-step unrolled gradient approximations, so no manual prompts are needed at inference.
Key results
- With 4 labeled examples on CelebAMask, BLO-SAM reaches 65.9% overall Dice versus 62.9% for Med-SA (which needs manual prompts) and 58.2% for SAMed.
- On car components with 2 examples, BLO-SAM gets 71.1% overall Dice versus 67.3% for Med-SA and 64.1% for SSP.
- On Kvasir with 4 examples, BLO-SAM achieves 59.7% Dice versus 42.7% for SAMed and 33.5% for Med-SA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bi-level optimization to train model weights on D1 and prompt embeddings on disjoint D2 instead of jointly optimizing both on the combined data (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Helped |
| Use second-order rather than first-order gradients for prompt embedding optimization (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Helped |
| Finetune all three SAM components rather than only the mask decoder (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Helped |
| Finetune the image encoder with LoRA (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Mixed |
| Finetune the prompt encoder convolution layers (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Mixed |
| Set LoRA rank to 4 rather than other tested ranks (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Helped |
| Fully finetune all SAM parameters instead of using BLO-SAM (ablation) | BLO-SAM · Human body, 4 examples | Dice score: 75.5% vs 47.5% | Hurt |
| Increase training examples from 4 to 512 (ablation) | BLO-SAM · Human body | Dice score | Helped |
| Increase the trade-off parameter λ toward a middle-ground setting rather than an extreme (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Helped |
| Use a very large LoRA rank (ablation) | BLO-SAM · Human body and teeth, 4 examples | Dice score | Hurt |
Surprising findings
- Full finetuning of SAM (FT-SAM) overfits severely on the body task, reaching only 47.5% Dice versus 75.5% for BLO-SAM.
- Vanilla SAM with automatically constructed ground-truth-derived prompts scores only 32.9% Dice on CelebAMask with 4 examples.
Limitations
- Baselines Med-SA and SAM use prompts derived from ground-truth masks of test images, a setting the authors acknowledge is impractical.
- Second-order bi-level gradient is more costly; main method uses first-order approximation.
- Experiments use only very small training sets (mostly under 10 examples) and a ViT-B backbone.
Experimental setup
Datasets: CelebAMask-HQ Car segmentation (Kaggle) TikTok dances human body segmentation Children's dental panoramic radiographs Kvasir-SEG JSRT lung
Benchmarks: CelebAMask-HQ Car segmentation TikTok dances body Children's dental panoramic radiographs (teeth) Kvasir-SEG JSRT lung
Metrics: Dice score
Baselines: DeepLabV3 SwinUnet HSNet SSP SAM Med-SA SAMed FT-SAM
Builds on: Segment Anything Model LoRA SAMed Med-SA DARTS-style unrolled bi-level optimization
Closest papers by meaning
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.90)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.88)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.88)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.88)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.