Research map / Adversarial robustness and security / Attacks on text and graphs
ASAM: Boosting Segment Anything Model with Adversarial Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASAM, which fine-tunes SAM's output and mask tokens on natural adversarial images generated by Stable Diffusion latent optimization plus a mask-conditioned ControlNet, using 1% of SA-1B.
Problem
SAM underperforms on niche downstream segmentation distributions, and existing fixes (fine-tuning, adapters, post-processing) hurt generalization or add parameters and annotated data; the paper asks how to boost SAM's generalization without extra data, architecture changes, or losing zero-shot abili
Approach
Adapt NLP-style adversarial tuning to SAM by generating "natural" adversarial examples: DDIM-invert SA-1B images into Stable Diffusion latents, optimize the latent adversarially against SAM's segmentation loss, and re-generate images with a mask-conditioned ControlNet so they stay aligned with the original masks; then fine-tune only SAM's output and mask tokens (~0.001% of parameters).
Key results
- ASAM improves over zero-shot SAM on all 14 box-prompt datasets, averaging 77.6 vs 75.8 mIoU (+1.3 avg, with +1.5 on several sets).
- Baselines PGD tuning (56.2 avg) and DAT tuning (51.1 avg) and DatasetDM (44.0 avg) degrade SAM substantially on the same 14 datasets.
- Applying the framework to EfficientSAM (vit-tiny) raises average results on five datasets, e.g. Cityscapes 48.7 to 49.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SAM using natural adversarial examples generated from 1% of SA-1B | SAM · 14-dataset evaluation (first group) | average mIoU: 76.7 -> 77.6 | Helped |
| Fine-tune SAM using natural adversarial examples generated from 1% of SA-1B | SAM · 14-dataset evaluation (second group) | average mIoU: 75.8 -> 77.5 | Helped |
| Use latent projection, latent optimization, and controllable generation together (ablation) | ASAM · 14-dataset evaluation | mIoU: 59.1 -> 77.6 | Helped |
| Add latent optimization to latent projection while omitting controllable generation (ablation) | ASAM · 14-dataset evaluation | mIoU: 59.1 -> 54.3 | Hurt |
| Add controllable generation to latent projection while omitting latent optimization (ablation) | ASAM · 14-dataset evaluation | mIoU: 59.1 -> 69.3 | Helped |
| Add latent optimization when controllable generation is included (ablation) | ASAM · 14-dataset evaluation | mIoU: 69.3 -> 77.6 | Helped |
Surprising findings
- Standard adversarial-style tuning (PGD, DAT) and DatasetDM-generated data substantially hurt SAM's segmentation performance across most datasets, despite SAM being trained on large data.
Limitations
- Theoretical underpinnings are acknowledged as unexplored; the link to prior theory is only suggestive.
- Gains are modest (about 1.3 mIoU average over SAM).
- Evaluation relies on a self-compiled suite of 14 datasets with box prompts.
Experimental setup
Datasets: SA-1B (sa_000000 subset) LAION-5B
Benchmarks: DOORS LVIS ZeroWaste-f NDISPark Egohos Plittersdorf BBC038V1 Ade20k VOC2012 Cityscapes COCO2017 HRSOD-TE
Metrics: mIoU NIMA-AVA HyperIQA MUSIQ-AVA TReS
Baselines: SAM SAM + PGD Tuning SAM + DAT Tuning SAM + DatasetDM EfficientSAM
Builds on: Segment Anything Model DDIM inversion null-text inversion ControlNet Stable Diffusion Projected gradient descent adversarial training
Closest papers by meaning
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.88)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.86)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.86)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.