Research map / Architectures and efficiency / Pruning and efficient tuning
SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-PARSER, which fine-tunes SAM's encoder by SVD-decomposing its convolutional-layer weights and training only the diagonal singular-value coefficients (~0.5k params), reducing trainable parameters ~290x versus LoRA while matching or beating PEFT baselines on most datasets.
Problem
Parameter-efficient fine-tuning of the Segment Anything Model for diverse downstream segmentation scenarios typically adds new trainable parameters; the paper asks whether SAM can be adapted with nearly zero new trainable parameters.
Approach
Assume SAM's original parameter space is complete enough that its SVD bases can reconstruct task-specific weight matrices; freeze the bases (U, V) and fine-tune only the diagonal coefficient matrix P, so adaptation reuses existing weights rather than adding new parameter subspaces.
Key results
- Reduces trainable parameters ~290x relative to LoRA (0.5k coefficients vs. LoRA's added parameters) while reaching 84.2% mIoU on NWPU versus 83.9% for LoRA.
- On medical CT abdominal organ segmentation, improves DSC by 0.8% over LoRA (91.1% vs 90.3%).
- On SSDD radar images, SAM-PARSER (81.6 mIoU) falls behind LoRA (82.4) and Adapter (81.8), which the authors attribute to SAM's parameter space not covering very different image domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use convolution-layer parameter space only for reconstruction instead of Transformer-layer parameter space (ablation) | SAM-PARSER · NWPU VHR-10 | mIoU (%): 83.1 -> 84.2 | Helped |
| Use convolution-layer parameter space only instead of both Transformer- and convolution-layer spaces (ablation) | SAM-PARSER · NWPU VHR-10 | mIoU (%): 83.3 -> 84.2 | Helped |
| Add bases to coefficient-only fine-tuning (ablation) | SAM-PARSER · NWPU VHR-10 | mIoU (%): 84.2 -> 85.3 | Helped |
| Fine-tune coefficients instead of bases only (ablation) | SAM-PARSER · NWPU VHR-10 | mIoU (%): 84.9 -> 84.2 | Hurt |
| Fine-tune coefficients instead of both bases and coefficients (ablation) | SAM-PARSER · NWPU VHR-10 | mIoU (%): 85.3 -> 84.2 | Hurt |
| Apply SAM-PARSER rather than LoRA | SAM ViT-Base · COCO2017 val | mIoU (%): 75.3 -> 75.0 | Hurt |
| Apply SAM-PARSER rather than LoRA | SAM ViT-Base · CT Abdominal organ test | DSC (%): 90.1 -> 90.9 | Helped |
| Apply SAM-PARSER rather than LoRA | SAM ViT-Base · NWPU VHR-10 val | mIoU (%): 83.9 -> 84.2 | Helped |
| Apply SAM-PARSER rather than LoRA | SAM ViT-Base · WHU building extraction val | mIoU (%): 81.9 -> 81.8 | No clear effect |
| Apply SAM-PARSER rather than LoRA | SAM ViT-Base · SSDD val | mIoU (%): 82.4 -> 81.6 | Hurt |
Surprising findings
- Fine-tuning only convolutional-layer coefficients outperforms using transformer-layer parameters for reconstruction (84.2 vs 82.9-83.3 mIoU on NWPU).
- On the large COCO2017 dataset, SAM-PARSER is slightly worse than LoRA (0.2% mIoU), which the authors attribute to overlap with SAM's original parameter space.
Limitations
- Fails to match LoRA/Adapter on the SSDD radar dataset, where the target image domain is distant from natural images.
- Gains are small on several datasets (e.g., 0.2-0.4% mIoU on VOC/COCO-level results), and COCO results are slightly below LoRA.
- Relies on the assumption that SAM's original parameter space is complete enough to cover new scenarios.
Experimental setup
Datasets: CT Abdominal organ COCO2017 PASCAL VOC2012 NWPU VHR-10 WHU building extraction
Benchmarks: CT Abdominal organ COCO2017 PASCAL VOC2012 NWPU VHR-10 WHU building extraction SSDD
Metrics: DSC mIoU F1 Params (K) Fps
Baselines: LoRA Adapter FacT decoder-only fine-tuning
Builds on: Segment Anything Model LoRA Adapter FacT SAMed
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.92)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- Personalize Segment Anything Model with One Shot (ICLR 2024 · similarity 0.88)
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.88)
- BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation (ICML 2024 · similarity 0.88)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.