Research map / Image generation / Text-to-image diffusion

StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations

NeurIPS 2025 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StyleGuard, an adversarial perturbation method that disrupts style-related latent statistics and adds an upscale loss over ensemble purifiers/upscalers to protect images against DreamBooth and Textual Inversion style mimicry, including under DiffPure and Noise Upscaling.

Problem

Artists' images can be used to fine-tune text-to-image diffusion models for style mimicry; existing protective perturbations are fragile to diffusion-based purification (DiffPure, Noise Upscaling) and transfer poorly across models.

Approach

A latent-space style loss that shifts per-channel mean and variance statistics of protected images away from the originals toward a distinct-style target across multiple substitute encoders, combined with an upscale loss that maximizes denoising error on ensemble purifiers/upscalers, optimized with alternating bilevel PGD and unrolled fine-tuning.

Key results

• On WikiArt with DreamBooth and no preprocessing, StyleGuard reaches FID 428.70 and precision 0.00, versus 233.78/0.60 for clean images and the best baseline SimAC at 407.40/0.00.
• Under Noise Upscale, StyleGuard achieves FID 401.80 and precision 0.00, while MetaCloak, SimAC, and Mist reach precision 0.40, 0.45, and 0.45 respectively.
• On LoRA fine-tuning of SD XL, StyleGuard gets FID 464.45 and precision 0.00, though on SD v2-1 LoRA it reaches only precision 0.16 versus 0.45 for Anti-DreamBooth.

Measured effects reported in the paper

Surprising findings

• StyleGuard is less effective under LoRA fine-tuning on SD v2.1 (precision 0.16) than on SD XL, which the authors attribute to parameter adaptation differences.
• The style loss alone degrades image quality for unprotected images, and the denoise-only variant is weaker against Noise Upscale than the full method.

Limitations

• Less effective against commercial upscalers and LoRA-based fine-tuning, per the authors.
• Computationally heavier: about 20 minutes per image on eight 3090 GPUs, versus 15 minutes for MetaCloak.
• Evaluated mainly on SD-family models and a limited set of purifiers and upscalers.

Experimental setup

Datasets: WikiArt CelebA

Benchmarks: WikiArt style mimicry (FID, precision); CelebA identity matching (IMS)

Metrics: FID precision mimicry success rate identity match score

Baselines: Glaze Mist Anti-DreamBooth MetaCloak SimAC

Builds on: Anti-DreamBooth MetaCloak Mist DreamBooth Noise Upscaling

Closest papers by meaning

• Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI (ICLR 2025 · similarity 0.89)
• DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.88)
• Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models (ICLR 2025 · similarity 0.86)
• Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.85)
• BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing (NeurIPS 2025 · similarity 0.85)
• GuardT2I: Defending Text-to-Image Models from Adversarial Prompts (NeurIPS 2024 · similarity 0.85)