Research map / Image generation / GANs
Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Attacks pre-trained deterministic conditional image generators (e.g., LaMa, StyTr^2, SwinIR) with PGD-like micro perturbations on the input condition, using L1/variance losses for diversity and CLIP directional losses for text/image-guided control, without retraining.
Problem
Pre-trained deterministic conditional image generative models produce a single fixed output per input, which is limiting for subjective tasks like large-mask inpainting and style transfer; existing diversity methods require retraining, complex noise injection, or transformation functions.
Approach
Treat diversity as an adversarial attack: add small PGD-style perturbations to the input condition of a frozen deterministic generator, with untargeted statistical losses for diversity and a CLIP directional loss as a targeted attack to steer outputs by text or image references.
Key results
- On inpainting, LaMa+Ours reaches LPIPS 0.0606 and L1 13.60 diversity versus MAT's 0.0176 and 6.00, with FID 52.98 vs LaMa's 50.37.
- On style transfer, StyTr^2+Ours reaches LPIPS 0.3199 and L1 51.67, exceeding epsilon-AE (0.1826) and CNNMRF+DivSwapper (0.2183).
- Each 10-step attack takes about 1.1 to 2 seconds on a single RTX 3090, versus roughly 5.5 s for StableDiffusion-v1.5 and 89 s for ICT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a 10-step adversarial attack to LaMa | LaMa · Inpainting | LPIPS: 0 -> 0.0606 | Helped |
| Add a 10-step adversarial attack to LaMa | LaMa · Inpainting | FID: 50.37 -> 52.98 | Hurt |
| Add a 10-step adversarial attack to StyTr² | StyTr² · Style transfer | LPIPS: 0 -> 0.3199 | Helped |
| Add a 10-step adversarial attack to StyTr² | StyTr² · Style transfer | L1: 0 -> 51.6725 | Helped |
| Use noise truncation instead of PGD truncation (ablation) | Untargeted adversarial attack · Conditional image generation | Visual plausibility | Helped |
| Use optimizer-based updates instead of the proposed attack | LaMa inpainting · Inpainting | L1 distance between updated perturbations: 3.9×10⁻³ for our method; 2.54×10⁻⁵ for optimizer-based method | Hurt |
| Use optimizer-based updates instead of the proposed attack | LaMa inpainting · Inpainting | Generation quality | Hurt |
Surprising findings
- Even well-posed super-resolution models (SwinIR) can be attacked to produce sharper, more detailed outputs.
- CLIP similarity and human perception are not strictly aligned: over-attacked outputs can score higher on CLIP text similarity while looking less like the target.
Limitations
- Large attack strength can produce artifacts and meaningless outputs, and CLIP guidance can overfit the text rather than match human perception.
- Diversity is bounded by the generator's learned mapping, e.g., BigGAN produces off-label samples under large perturbations.
- Attack hyperparameters (epsilon, clip bounds) must be tuned per task.
Experimental setup
Benchmarks: LaMa inpainting test setup StyTr^2 style transfer setup
Metrics: LPIPS L1 distance FID runtime
Baselines: PIC MAT StableDiffusion-v1.5 ICT BAT epsilon-AE DivSwapper Avatar-net+DFP CLIPstyler optimizer-based (Adam) method
Builds on: FGSM PGD CLIP directional CLIP loss StyleGAN-NADA
Closest papers by meaning
- Diversity-Sensitive Conditional Generative Adversarial Networks (ICLR 2019 · similarity 0.91)
- Structured Generative Adversarial Networks (NeurIPS 2017 · similarity 0.89)
- Robust Conditional Generative Adversarial Networks (ICLR 2019 · similarity 0.88)
- Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space (CVPR 2017 · similarity 0.88)
- CcGAN: Continuous Conditional Generative Adversarial Networks for Image Generation (ICLR 2021 · similarity 0.88)
- Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.