Research map / Adversarial robustness and security / Attacks on text and graphs

Adaptive Median Smoothing: Adversarial Defense for Unlearned Text-to-Image Diffusion Models at Inference Time

ICML 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Diffusion-based adversarial purification

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Adaptive Median Smoothing, an inference-time defense that adds token-wise anisotropic Gaussian noise to text embeddings (scaled by concept relevance) and takes an element-wise median, to make unlearned Stable Diffusion models robust to adversarial prompts.

Problem

Unlearned text-to-image diffusion models can be prompted with adversarial inputs that regenerate erased concepts such as nudity and violence, and existing defenses either hurt utility or are costly at inference.

Approach

Reformulates robustness of unlearned diffusion as robust regression and generalizes median smoothing from isotropic to anisotropic Gaussian noise with a provable ellipsoidal guarantee; noise intensity per token is set by its cosine relevance to a target-concept direction, and the median is taken over text embeddings (not noise predictions) to save compute.

Key results

• On ESD-unlearned SD v1.4 for nudity, the method reaches average ASR 6.80% with FID 7.365, versus 25.42% ASR undefended and 5.89% ASR with SLD-Max (FID 25.544).
• On UCE-unlearned model, the method achieves average ASR 11.02% with FID 5.343, outperforming RECE (14.36% ASR, FID 6.001).
• Inference time is 10.32 s per image versus 11.78 s for SLD and 9.44 s undefended, on V100 GPUs.

Measured effects reported in the paper

Surprising findings

• Simple blacklist-based word removal barely reduces ASR for UCE-unlearned models (30.63% vs 40.34% undefended on average), and SLD at max settings lowers ASR but badly degrades FID.

Limitations

• Failure cases exist where some adversarial prompts bypass the defense at default sigma0; larger sigma0 hurts benign utility.
• Requires positive and negative prompt pairs to compute the concept direction.
• Efficiency decreases with many concepts in multi-concept unlearning.

Experimental setup

Datasets: ViSU prompt pairs (7,863 nudity pairs) COCO validation set

Benchmarks: I2P P4D UDA RAB MMA QF-PGD

Metrics: attack success rate (ASR) FID CLIP score inference time

Baselines: RACE RECE SLD-Weak SLD-Medium SLD-Strong SLD-Max Blacklist POSI

Builds on: Randomized smoothing median (Chiang et al., 2020) ESD UCE Safe Latent Diffusion

Closest papers by meaning

• Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models (NeurIPS 2024 · similarity 0.88)
• Robust Representation Consistency Model via Contrastive Denoising (ICLR 2025 · similarity 0.88)
• Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.87)
• STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)
• CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models (ICML 2025 · similarity 0.87)
• The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)