Research map / Detection and segmentation / Open-set and few-shot detection
Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conv-LoRA, a parameter-efficient finetuning method that adds MoE-gated multi-scale convolutions inside LoRA's bottleneck to inject local priors into SAM's ViT encoder, and adapts SAM to multi-class segmentation.
Problem
SAM underperforms on specialized domains (medical, remote sensing, agriculture, natural images); its plain ViT lacks vision-specific inductive biases and its binary foreground-background pretraining hampers high-level semantic learning, so efficient adaptation is needed.
Approach
Insert lightweight 3x3 convolutions between LoRA's down- and up-projections, organized as a mixture of experts where each expert convolves at a different feature scale (via interpolation), with top-k gating choosing scales per input; also add a classification branch to SAM's mask decoder for end-to-end multi-class segmentation.
Key results
- On binary segmentation, Conv-LoRA (4.02M params, 0.63%) achieves the best score on all reported datasets, e.g., 77.6 Jaccard on ISIC 2017 vs. 76.6 for LoRA and 2.54 BER on SBU vs. 2.74 for LoRA.
- On Trans10K-v2 (12-class), Conv-LoRA reaches 95.07 accuracy and 67.09 mIoU versus 94.80 / 66.01 for LoRA with the same parameter budget.
- Linear probing on ImageNet-1K shows SAM's ViT-B encoder reaches 54.2% accuracy vs. 67.7% for MAE, suggesting segmentation pretraining hurts high-level semantics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add lightweight convolutional experts to LoRA | SAM image encoder · ISIC 2017 | Jac: 76.6 ± 0.23 -> 77.6 ± 0.57 | Helped |
| Use dynamic MoE expert selection instead of simultaneous multi-scale feature fusion (ablation) | Conv-LoRA · ISIC 2017 | Jac: 77.4 -> 77.9 | Helped |
| Use dynamic MoE expert selection instead of simultaneous multi-scale feature fusion (ablation) | Conv-LoRA · Training | Training Speed (Iter / s): 0.79 -> 1.22 | Helped |
| Use one expert with scaling ratio 4 rather than ratios 1, 2, or 8 (ablation) | Conv-LoRA · Leaf | IoU: 73.6; 73.8; 74.0; 73.2 | Mixed |
| Use one expert with scaling ratio 2 rather than ratios 1, 4, or 8 (ablation) | Conv-LoRA · ISIC 2017 | Jac: 76.8; 77.3; 76.9; 76.9 | Mixed |
| Increase LoRA rank from 3 to 24 (ablation) | LoRA · Trans10K-v2 | mIoU: 66.01 -> 67.02 | Helped |
| Add VPT to Conv-LoRA (ablation) | Conv-LoRA · CVC-ColonDB | Sα: 85.1 -> 86.0 | Helped |
| Add VPT to Conv-LoRA (ablation) | Conv-LoRA · ETIS | Sα: 86.8 -> 87.7 | Helped |
Surprising findings
- SAM's image encoder linear-probe accuracy (54.2%) is markedly lower than its MAE initialization (67.7%), indicating segmentation pretraining degrades high-level semantic features.
- Decoder-only fine-tuning matches domain-specific methods on accuracy but lags badly in mIoU for multi-class segmentation, pointing to the encoder's limited semantic capacity.
Limitations
- Conv-LoRA may not consistently outperform domain-specific SOTA models on specific datasets.
- Multi-class performance on 12-class Trans10K-v2 still trails the fully trained Trans2Seg, and gains from more rank come at the cost of parameter efficiency.
- Optimal feature scale for local prior injection is not definitively determined.
Experimental setup
Datasets: Kvasir CVC-612 ISIC 2017 CAMO COD10K CHAMELEON SBU Leaf Disease Segmentation Massachusetts Roads Trans10K-v1 Trans10K-v2 ImageNet-1K
Benchmarks: Kvasir CVC-612 ISIC 2017 CAMO SBU Leaf Disease Segmentation Massachusetts Roads Trans10K-v1 Trans10K-v2 CHAMELEON COD10K CVC-ColonDB
Metrics: S-alpha E-phi F-beta weighted MAE BER IoU Dice Jaccard accuracy mIoU
Baselines: Decoder-only fine-tuning BitFit Adapter VPT LST SAM-Adapter SSF LoRA
Builds on: LoRA Segment Anything Model Mixture-of-Experts Mask2Former Convpass
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.87)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.86)
- GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation (NeurIPS 2025 · similarity 0.86)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.85)
- LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.