Research map / Adversarial robustness and security / Attacks on text and graphs
Robust SAM: On the Adversarial Robustness of Vision Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a cross-prompt adversarial attack that works on both point and box prompts for SAM and SAM 2, and RobustSAM, a defense that trains only 512 singular values of convolutional layers via SVD to recover segmentation under attack.
Problem
Adversarial vulnerability of vision foundation models like SAM and SAM 2, where existing attacks do not transfer across prompt types and existing parameter-efficient defenses trade off clean accuracy for robustness.
Approach
A cross-prompt attack targets key feature channels shared by point- and box-prompt decoders (intersection of top-K channels by output difference), and a defense that adapts only the diagonal singular-value matrix P of SVD-decomposed convolutional layers in SAM's encoder, keeping U and V frozen.
Key results
- Cross-prompt attack reaches 60.2%, 62.9%, and 75.6% ASR on COCO, VOC, and SA1B for SAM at epsilon=16/255, outperforming PPA and BPA.
- RobustSAM adapts only 512 parameters (0.5K) and reaches 35.0 mIoU on VOC under the cross-prompt attack versus 6.4 for the undefended baseline and 27.0 for SAM-LoRA.
- On clean VOC box prompts, RobustSAM keeps 80.9 mIoU versus 87.1 for baseline, while SAM-LoRA drops to 63.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cross-prompt attack instead of BPA | SAM · COCO, ε=16/255 | ASR: 35.7 -> 60.2 | Helped |
| Use cross-prompt attack instead of BPA | SAM · VOC, ε=16/255 | ASR: 44.4 -> 62.9 | Helped |
| Use cross-prompt attack instead of BPA | SAM · SA1B, ε=16/255 | ASR: 44.5 -> 75.6 | Helped |
| Use cross-prompt attack instead of BPA | SAM 2 · COCO, ε=16/255 | ASR: 18.5 -> 32.8 | Helped |
| Use cross-prompt attack instead of BPA | SAM 2 · VOC, ε=16/255 | ASR: 20.0 -> 34.4 | Helped |
| Use cross-prompt attack instead of BPA | SAM 2 · SA1B, ε=16/255 | ASR: 21.5 -> 36.9 | Helped |
| Adapt convolutional-layer singular values with SVD | RobustSAM · VOC, CPA attack | mIoU: 30.4 -> 35.0 | Helped |
| Adapt convolutional-layer singular values with SVD | RobustSAM · COCO, CPA attack | mIoU: 28.1 -> 29.6 | Helped |
| Adapt convolutional-layer singular values with SVD | RobustSAM · SA1B, CPA attack | mIoU: 34.8 -> 36.0 | Helped |
| Adapt convolutional-layer singular values with SVD | RobustSAM · DAVIS, CPA attack | mIoU: 33.7 -> 37.9 | Helped |
| Adapt convolutional-layer singular values with SVD rather than SAM-LoRA | SAM · SA1B, BPA attack | mIoU: 44.3 -> 41.5 | Hurt |
| Adapt convolutional-layer singular values with SVD rather than SAM-LoRA | SAM · SA1B, PPA attack | mIoU: 57.5 -> 52.2 | Hurt |
Surprising findings
- Robustness depends more on which parameters are adapted than on how many: MedSAM, which adapts encoder and decoder with about 93.7M parameters, underperforms parameter-light methods.
- Under BPA and PPA on SA1B, SAM-LoRA slightly outperforms RobustSAM in mIoU.
Limitations
- Attack success rate on SAM 2 is considerably lower (32.8% to 36.9% at epsilon 16/255), indicating limited attack strength there.
- Under BPA and PPA on SA1B, SAM-LoRA slightly outperforms RobustSAM.
- Evaluation uses only 2000 sampled images with single-object ground truth masks.
Experimental setup
Datasets: SA1B VOC COCO DAVIS
Benchmarks: SA1B VOC COCO DAVIS
Metrics: attack success rate mIoU
Baselines: PPA BPA MedSAM SAM-Adapter SAM-LoRA SAM-COBOT FacT
Builds on: PGD LoRA SAM-Adapter SAMed FacT
Closest papers by meaning
- Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 (NeurIPS 2025 · similarity 0.90)
- Transferable Adversarial Attacks on SAM and Its Downstream Models (NeurIPS 2024 · similarity 0.90)
- DarkSAM: Fooling Segment Anything Model to Segment Nothing (NeurIPS 2024 · similarity 0.89)
- On the Duality Between Sharpness-Aware Minimization and Adversarial Training (ICML 2024 · similarity 0.88)
- ASAM: Boosting Segment Anything Model with Adversarial Tuning (CVPR 2024 · similarity 0.88)
- RAMP: Boosting Adversarial Robustness Against Multiple $l_p$ Perturbations for Universal Robustness (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.