Research map / Adversarial robustness and security / Certified and empirical robustness
Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UAP-SAM2, a cross-prompt universal adversarial perturbation attack against the SAM2 video/image segmentation model, using target-scanning over prompt regions and a dual semantic deviation loss (semantic confusion, feature shift, memory misalignment).
Problem
Existing adversarial attacks designed for SAM transfer poorly to SAM2 because of prompt-directed guidance and memory-based semantic entanglement across frames; robustness of SAM2 to universal, cross-prompt attacks is unexplored.
Approach
A target-scanning strategy assigns random prompts to k/m regions of each frame to reduce prompt dependency, and a dual semantic deviation objective perturbs image-encoder features within the current frame while breaking consistency between consecutive frames' features to disrupt SAM2's memory bank.
Key results
- On DAVIS with point prompts, UAP-SAM2 reduces SAM2 mIoU by over 45.79% (UAP) and 54.77% (sample-wise variant).
- In the UAP comparison on video (point prompts), UAP-SAM2 achieves the lowest mIoU on all three video datasets, e.g., 37.03 on YouTube vs. 49.39 for UAD and 67.51 for DarkSAM (SAM2-T).
- Under 4/255 epsilon, the attack still lowers average mIoU by over 33.08%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UAP-SAM2 instead of DarkSAM | SAM2-T · DAVIS video, point prompts | mIoU (%): 57.00 -> 42.47 | Helped |
| Remove semantic confusion loss (A) (ablation) | UAP-SAM2 · DAVIS | mIoU | Hurt |
| Remove feature shift loss (B) (ablation) | UAP-SAM2 · DAVIS | mIoU | Hurt |
| Remove memory misalignment loss (C) (ablation) | UAP-SAM2 · DAVIS | mIoU | Hurt |
| Vary target-scanning region count; use 256 regions (ablation) | UAP-SAM2 · DAVIS | mIoU: m = 256 | Helped |
| Increase optimization iterations to 10 (ablation) | UAP-SAM2 · DAVIS | mIoU: 10 | Helped |
| Increase perturbation budget (ablation) | UAP-SAM2 · DAVIS | mIoU | Helped |
| Vary negative sample count (ablation) | UAP-SAM2 · DAVIS | mIoU: 10 to 100 | Mixed |
| Evaluate with 15 frames per video rather than all frames (ablation) | UAP-SAM2 · DAVIS | mIoU: 15 frames | No clear effect |
Surprising findings
- Attacking only the first frame with DarkSAM fails to significantly degrade SAM2 even at epsilon 32/255, unlike on SAM.
- Model pruning offers limited defense: adversarial mIoU stays stable while benign mIoU drops as pruning ratio increases.
Limitations
- May not generalize directly to traditional segmentation models whose outputs are not label-free masks.
- Transferability to other segmentation frameworks is unclear.
Experimental setup
Datasets: YouTube-VOS2018 DAVIS 2017 MOSE
Benchmarks: YouTube-VOS2018 DAVIS 2017 MOSE YouTube* DAVIS* MOSE*
Metrics: mIoU
Baselines: PGD Attack-SAM S-RA UAD DarkSAM UAPGD SegPGD VOSPGD
Builds on: Segment Anything Model 2 Attack-SAM DarkSAM PGD UAD
Closest papers by meaning
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.90)
- DarkSAM: Fooling Segment Anything Model to Segment Nothing (NeurIPS 2024 · similarity 0.89)
- Transferable Adversarial Attacks on SAM and Its Downstream Models (NeurIPS 2024 · similarity 0.87)
- Data-free Universal Adversarial Perturbation with Pseudo-semantic Prior (CVPR 2025 · similarity 0.86)
- AAKR: Adversarial Attack-based Knowledge Retention for Continual Semantic Segmentation (AAAI 2025 · similarity 0.86)
- ASAM: Boosting Segment Anything Model with Adversarial Tuning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.