Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-COBOT, a plug-and-play PEFT add-on for SAM that adds cross-block coefficient communication via a learnable relation matrix and intra-block hyper-complex projection heads, adding about 1K parameters.
Problem
Existing PEFT methods inject few parameters per block, so they make only small adjustments to projection directions when adapting SAM to segmentation tasks, which need heavier changes than classification.
Approach
Treat per-block PEFT coefficient sets as a tensor and mix them across blocks with a learnable relation matrix (T-product-inspired inter-block communication), and add a hyper-complex (Hamilton-product) layer that generates a projection head's weights to enhance intra-block adjustment; the module plugs into LoRA and AdaptFormer.
Key results
- Adding SAM-COBOT to AdaptFormer raises the 10-dataset average from 82.4 to 83.6 mIoU/DSC with about 1.3K extra parameters (322.7K to 324.0K).
- Adding SAM-COBOT to LoRA raises the average from 81.1 to 81.8 with about 0.9K extra parameters.
- Ablation on ADOME shows the full module lifts AdaptFormer from 90.1 to 91.3 DSC, and a learnable relation matrix beats a fixed random one (91.3 vs 90.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAM-COBOT to Adaptformer | SAM ViT-Base · ADOME | DSC: 90.1 ± 0.2 -> 91.3 ± 0.5 | Helped |
| Add SAM-COBOT to Adaptformer | SAM ViT-Base · NWPU | mIoU: 83.0 ± 0.1 -> 84.0 ± 0.3 | Helped |
| Add SAM-COBOT to Adaptformer | SAM ViT-Base · TRCAN | mIoU: 73.3 ± 0.1 -> 74.1 ± 0.0 | Helped |
| Add SAM-COBOT to LoRA | SAM ViT-Base · BRAST | DSC: 84.8 ± 0.6 -> 85.3 ± 1.2 | Helped |
| Add dual coefficient sets (ablation) | Adaptformer · ADOME | DSC: 90.1 -> 90.1 (+0.0) | No clear effect |
| Add hyper-complex layer without CoS or RM (ablation) | Adaptformer · ADOME | DSC: 90.1 -> 90.3 (+0.2) | Helped |
| Add relation matrix with CoS, without HL (ablation) | Adaptformer · ADOME | DSC: 90.1 -> 90.9 (+0.8) | Helped |
| Add CoS, RM, and HL (ablation) | Adaptformer · ADOME | DSC: 90.1 -> 91.3 (+1.2) | Helped |
| Learn the relation matrix by back-propagation instead of using fixed random values (ablation) | Adaptformer+SAM-COBOT · ADOME | DSC: 90.2 ± 0.6 -> 91.3 ± 0.5 | Helped |
| Replace a linear layer with a hyper-complex layer (ablation) | Adaptformer+SAM-COBOT · ADOME | DSC: 90.9 ± 0.6 -> 91.3 ± 0.5 | Helped |
Limitations
- Gains are modest on several datasets (often under 1 point average improvement).
- Evaluated only with ViT-Base and ViT-Large SAM backbones and two PEFT base methods.
Experimental setup
Datasets: COCO TRCAN NWPU SSDD SONAR ADOME SPLEN MOMO BRAST SEGRAP
Benchmarks: COCO TRCAN NWPU SSDD SONAR ADOME SPLEN MOMO BRAST SEGRAP
Metrics: mIoU DSC
Baselines: LoRA AdaptFormer Lightweight mask-decoder tuning Freeze
Builds on: LoRA AdaptFormer Segment Anything Model T-product tensor algebra
Closest papers by meaning
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.89)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.88)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.88)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.87)
- InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective (ICML 2025 · similarity 0.87)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.