Research map / Adversarial robustness and security / Attacks on text and graphs
Unsegment Anything by Simulating Deformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UAD, a prompt-agnostic adversarial attack that optimizes a flow-field image deformation as the target for feature-space perturbation, so promptable segmentation models (SAM family, FastSAM) fail across prompts and models.
Problem
Promptable segmentation foundation models can extract any object from any image with a single prompt, raising copyright and misuse risks; the goal is an adversarial perturbation that disrupts segmentation for any prompt and transfers across model parameterizations.
Approach
Targeted feature attacks toward a learned deformed image (optimized via flow fields with SSIM-based deformation, smoothness/variance control, and a feature-fidelity term) transfer better than untargeted feature disruption or prompt-specific attacks, because the target stays near the natural image manifold.
Key results
- UAD on SAM-B (white-box) reduces mIoU to 51.53 with ASR@50 of 43.89%, versus 61.06 mIoU and 32.48% for the best baseline AA.
- On black-box SAM-H, UAD achieves mIoU 68.96 and ASR@50 23.42%, compared with 72.87 and 19.13% for AA.
- Targeted feature attacks (AA, PATA) transfer better than untargeted ones (TAP, ILPD), and prompt-specific Attack-SAM-K transfers worst.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UAD instead of AA | SAM-B attack transferred to SAM-B · SAM-1B subset, SAM-B (white-box) | mIoU: 61.06 ± 32.33 -> 51.53 ± 34.00 | Helped |
| Use UAD instead of AA | SAM-B attack transferred to SAM-L · SAM-1B subset, SAM-L | mIoU: 70.70 ± 29.74 -> 66.07 ± 32.04 | Helped |
| Use UAD instead of AA | SAM-B attack transferred to SAM-H · SAM-1B subset, SAM-H | mIoU: 72.87 ± 28.61 -> 68.96 ± 30.87 | Helped |
| Use UAD instead of AA | SAM-B attack transferred to FastSAM · SAM-1B subset, FastSAM | mIoU: 32.64 ± 39.58 -> 28.83 ± 38.36 | Helped |
| Use targeted feature disruption rather than untargeted feature disruption | Promptable segmentation model attacks · SAM-1B subset | mIoU | Helped |
| Use prompt-specific Attack-SAM-K rather than prompt-agnostic feature perturbation | Promptable segmentation model attacks · Unseen prompts | IoU | Hurt |
| Remove deformation loss LD (ablation) | UAD deformation optimization · Deformed target generation | structural change | Hurt |
| Remove control loss LC (ablation) | UAD deformation optimization · Deformed target generation | attack effectiveness | Hurt |
| Remove fidelity loss LF (ablation) | UAD deformation optimization · Adversarial attack | attack effectiveness | Hurt |
| Increase proxy adversarial update steps Tf (ablation) | UAD deformation optimization · Deformed target feature similarity | relative feature similarity | Helped |
| Use UAD across different perturbation budgets ϵ (ablation) | Adversarial attacks · SAM-B evaluation | mIoU | Helped |
Surprising findings
- Untargeted feature-disruption attacks transfer poorly across models even with similar architectures and training data, while targeted feature attacks transfer well.
- Prompt-specific adversarial noise overfits to its prompt and generalizes poorly to unseen prompts.
Limitations
- Evaluation is restricted to a 1000-image subset of SAM-1B and to the SAM family plus FastSAM.
- The proxy-sample fidelity loss and multi-flow-field deformation add computational overhead.
Experimental setup
Datasets: SAM-1B
Benchmarks: SAM-1B
Metrics: mIoU ASR@50 ASR@10
Baselines: Attack-SAM-K TAP ILPD AA PATA PATA++
Builds on: Segment Anything Model Attack-SAM PATA Activation attack TAP ILPD
Closest papers by meaning
- DarkSAM: Fooling Segment Anything Model to Segment Nothing (NeurIPS 2024 · similarity 0.86)
- UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation (NeurIPS 2024 · similarity 0.85)
- Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 (NeurIPS 2025 · similarity 0.84)
- ASAM: Boosting Segment Anything Model with Adversarial Tuning (CVPR 2024 · similarity 0.84)
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.83)
- DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.