Research map / Detection and segmentation / Open-set and few-shot detection
DarkSAM: Fooling Segment Anything Model to Segment Nothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DarkSAM, a prompt-free universal adversarial perturbation attack that combines a semantic-decoupling spatial attack with a wavelet-based high-frequency texture distortion attack to make SAM and its variants segment nothing.
Problem
Segment Anything Model and its variants are vulnerable to adversarial attacks, but prior attacks are sample-wise or rely on labels; a single universal perturbation that disables prompt-guided segmentation across images and prompts has not been studied.
Approach
A shadow target strategy builds a semantic blueprint by merging SAM masks from k random prompts, then a spatial attack flips foreground and background mask semantics while a DWT-based frequency attack increases high-frequency (texture) dissimilarity and constrains low-frequency components, all optimized into one UAP.
Key results
- DarkSAM reduces SAM mIoU by more than 60% on average across 96 settings, per the paper.
- In the comparison study on SAM with point prompts, DarkSAM reaches ASR of 64.96 (ADE20K), 61.63 (COCO), 50.63 (CITYSCAPES), 77.07 (SA-1B), while UAP, UAPGD, SSP, SegPGD and Attack-SAM achieve below 14 ASR.
- With a perturbation budget of 6/255, the average ASR still exceeds 45%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · ADE20K, point→point | ASR (%): 4.85 -> 64.96 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · MS-COCO, point→point | ASR (%): 1.52 -> 61.63 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · CITYSCAPES, point→point | ASR (%): 13.20 -> 50.63 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · SA-1B, point→point | ASR (%): 10.04 -> 77.07 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · ADE20K, box→box | ASR (%): 0.97 -> 69.72 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · MS-COCO, box→box | ASR (%): 0.51 -> 76.03 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · CITYSCAPES, box→box | ASR (%): 2.22 -> 64.27 | Helped |
| Use DarkSAM instead of the strongest listed baseline | SAM ViT-B · SA-1B, box→box | ASR (%): 3.46 -> 84.22 | Helped |
| Add the foreground evasion, background manipulation, high-frequency distortion, and low-frequency constraint components (ablation) | DarkSAM · ADE20K | ASR (%) | Helped |
| Set the shadow-target prompt count to 10 rather than lower tested counts (ablation) | DarkSAM · ADE20K | ASR (%) | Helped |
| Increase the perturbation budget (ablation) | DarkSAM · ADE20K | ASR (%) | Helped |
| Use a perturbation budget of 6/255 (ablation) | DarkSAM · ADE20K | ASR (%): exceeding 45% | Helped |
Surprising findings
- Some baseline attacks yield negative ASR, which the authors attribute to perturbations making inputs resemble SAM's training distribution and thereby improving accuracy.
- Box-prompt UAPs transfer better than point-prompt UAPs across prompt types.
Limitations
- The method is tailored to prompt-guided label-free mask outputs and may not apply to traditional segmentation models.
- Adversarial examples could mislead SAM-based platforms in sensitive domains such as medical imaging.
- Only contrast and brightness corruptions were tested as defenses.
Experimental setup
Datasets: ADE20K MS-COCO CITYSCAPES SA-1B
Benchmarks: ADE20K MS-COCO CITYSCAPES SA-1B
Metrics: mIoU ASR
Baselines: UAP UAPGD SSP SegPGD Attack-SAM
Builds on: Segment Anything Model universal adversarial perturbation HQ-SAM PerSAM
Closest papers by meaning
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.89)
- BadSAM: Exploring Security Vulnerabilities of SAM via Backdoor Attacks (Student Abstract) (AAAI 2024 · similarity 0.89)
- Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 (NeurIPS 2025 · similarity 0.89)
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.87)
- Unsegment Anything by Simulating Deformation (CVPR 2024 · similarity 0.86)
- ASAM: Boosting Segment Anything Model with Adversarial Tuning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.