Research map / Architectures and efficiency / Pruning and efficient tuning
BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BA-SAM, a training-light modification to SAM's attention (a length-dependent scaling factor plus a distance-based additive bias) that reduces performance degradation when inference image resolution differs from the 1024 training resolution.
Problem
SAM's performance degrades when applied to images whose resolution differs from its 1024x1024 pretraining size, and prior fixes (resizing, patch-size changes, retraining) are costly or erase SAM's prior knowledge.
Approach
Reframes resolution variation as length extrapolation: a new scaling factor log_{N_train}(N_test)/sqrt(d_k) keeps attention entropy stable as token count grows, and a bias beta*|i-j| added to query-key scores penalizes distant tokens, both added without changing SAM's architecture.
Key results
- In the zero-shot setting on COD10K, BA-SAM (on MobileSAM) reduces the 4096-resolution MAE degradation from 16.5% to 19.4%-equivalent... specifically its 4096 MAE is 0.246 vs 0.300 for MobileSAM, with degradation 19.4% vs 16.5%, while the 1024-to-2048 degradation on DUTS falls from 6.5% to 5.4% for S
- With fine-tuning, BA-SAM (MobileSAM-based) achieves MAE 0.053 on ISIC, 0.031 on DUTS, 0.055 on DIS-TE4 and 0.029 on COD10K at 1024 resolution, outperforming the listed specialized methods such as SegMaR (0.034 on COD10K).
- On COCO zero-shot instance segmentation, BA-SAM improves SAM AP from 42.5 to 43.0 and MobileSAM AP from 40.8 to 41.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the new scaling factor and bias-mode attention mask | MobileSAM · ISIC | MAE: 0.059 -> 0.053 | Helped |
| Add the new scaling factor and bias-mode attention mask | MobileSAM · DUTS | MAE: 0.034 -> 0.031 | Helped |
| Add the new scaling factor and bias-mode attention mask | MobileSAM · DIS-TE4 | MAE: 0.057 -> 0.055 | Helped |
| Add the new scaling factor and bias-mode attention mask | MobileSAM · COD10K | MAE: 0.030 -> 0.029 | Helped |
| Add the new scaling factor and bias-mode attention mask | SAM · COCO | AP: 42.5 -> 43.0 | Helped |
| Add the new scaling factor and bias-mode attention mask | MobileSAM · COCO | AP: 40.8 -> 41.2 | Helped |
| Add the new scaling factor (ablation) | MobileSAM · ISIC, zero-shot, resolution variation | performance degradation (∆diff): 17.8 -> 16.4 | Helped |
| Add the bias-mode attention mask (ablation) | MobileSAM · ISIC, zero-shot, resolution variation | performance degradation (∆diff): 17.8 -> 16.8 | Helped |
| Add the new scaling factor (ablation) | MobileSAM · DUTS, zero-shot, resolution variation | performance degradation (∆diff): 6.5 -> 7.5 | Hurt |
| Add the bias-mode attention mask (ablation) | MobileSAM · DUTS, zero-shot, resolution variation | performance degradation (∆diff): 6.5 -> 7.2 | Hurt |
| Add the new scaling factor (ablation) | MobileSAM · ISIC, fine-tuned at 1024 | performance degradation (∆diff): 42.2 -> 40.9 | Helped |
| Add the bias-mode attention mask (ablation) | MobileSAM · ISIC, fine-tuned at 1024 | performance degradation (∆diff): 42.2 -> 41.2 | Helped |
Surprising findings
- The zero-shot BA-SAM variant on MobileSAM shows larger relative degradation at 4096 on COD10K (19.4%) than MobileSAM baseline's 16.5%, despite lower absolute MAE.
- Adding the new scaling factor alone slightly worsens COD10K degradation in the zero-shot setting (11.9% to 12.0%).
Limitations
- Evaluated only on segmentation/dense prediction datasets at a limited set of resolutions (up to 4096 for some tasks).
- Bias slope beta must be set manually (beta=1 zero-shot, 0.1 found best in fine-tuning), with limited analysis of sensitivity across tasks.
- Generalized model results on COD10K in Table 2 appear inconsistent with the text (0.054 vs. other values), raising reporting concerns.
Experimental setup
Datasets: DUTS DIS5K ISIC COD10K COCO
Benchmarks: DUTS DIS-TE4 ISIC COD10K COCO
Metrics: MAE AP
Baselines: SAM MobileSAM MSA FlexiViT CASS VST IS-Net SINet
Builds on: Segment Anything Model MobileSAM scaled dot-product attention ALiBi
Closest papers by meaning
- Stable Segment Anything Model (ICLR 2025 · similarity 0.89)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.88)
- EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (CVPR 2024 · similarity 0.87)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · similarity 0.87)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.