Research map / Image generation / Text-to-image diffusion
WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WOUAF, a distributor-side method that fine-tunes only the Stable Diffusion decoder with user-specific weight modulation so each downloaded model imprints a 32-bit user fingerprint that can be decoded from generated images for attribution.
Problem
Attributing responsibility for malicious text-to-image outputs: existing fingerprinting relies on separate post-hoc modules that are easily bypassed, or on per-user fine-tuning that scales linearly with users, and is fragile under image post-processing.
Approach
Modulate the weights of the Stable Diffusion decoder with a user-specific fingerprint via a mapping network and affine layers (StyleGAN2-style weight modulation), fine-tuning once so that a single forward pass produces a per-user model whose outputs carry a decodable fingerprint, with no structural change to the model.
Key results
- WOUAF-conv and WOUAF-all reach 0.99 attribution accuracy on MS-COCO vs 0.99 for Stable Signature and 0.70 for DAG, with FID 24.43/24.42 vs 24.48 for original SD.
- Under image post-processing, WOUAF improves attribution accuracy by 11% over Stable Signature and 29% over DAG on average.
- Fingerprinting per user takes under 1 second versus 8.4 hours for DAG and under 1 minute for Stable Signature, with a 1-million-user database producing no false attributions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WOUAF weight modulation instead of Stable Signature | Stable Diffusion · MS-COCO test | attribution accuracy: 0.99 -> 0.99 | No clear effect |
| use WOUAF weight modulation instead of DAG | Stable Diffusion · MS-COCO test | attribution accuracy: 0.70 -> 0.99 | Helped |
| use WOUAF weight modulation instead of DAG | Stable Diffusion · LAION-Aesthetics | attribution accuracy: 0.71 -> 0.99 | Helped |
| use WOUAF-all instead of Stable Signature | Stable Diffusion · LAION-Aesthetics | FID: 20.02 -> 19.91 | Helped |
| use WOUAF-all instead of Stable Signature | Stable Diffusion · LAION-Aesthetics | CLIP-score: 0.50 -> 0.51 | Helped |
| set fingerprint dimension to 128 (ablation) | WOUAF-conv · Attribution accuracy | attribution accuracy: 0.99 -> 0.94 | Hurt |
| set fingerprint dimension to 128 (ablation) | WOUAF-all · Attribution accuracy | attribution accuracy: 0.99 -> 0.97 | Hurt |
| modulate weights in both the diffusion model and decoder rather than the decoder alone (ablation) | WOUAF · Image generation | CLIP-score: 0.68 | Hurt |
| train for robustness to image post-processing | WOUAF · Image post-processing robustness | attribution accuracy: 11% | Helped |
| compress generated images with auto-encoders | WOUAF · Auto-encoder fingerprint removal | attribution accuracy | Hurt |
| fine-tune the decoder to remove the fingerprint | WOUAF · Model purification | attribution accuracy | Hurt |
Surprising findings
- Modulating the diffusion U-Net in addition to the decoder hurt quality or attribution: the best such variant reached only 89% accuracy with FID 63.48.
Limitations
- Attribution accuracy degrades toward chance (~50%) under strong auto-encoder compression, at the cost of image quality.
- Model purification by fine-tuning the decoder also reduces attribution accuracy but degrades FID.
- Evaluated only on Stable Diffusion and image modality; extension to text, audio, video is future work.
Experimental setup
Datasets: MS-COCO LAION-Aesthetics ProGAN/StyleGAN/StyleGAN2 images
Benchmarks: MS-COCO test set LAION-Aesthetics
Metrics: attribution accuracy CLIP-score FID p-value of bit-match test binary classifier accuracy
Baselines: DAG Stable Signature Original SD
Builds on: StyleGAN2-ADA weight modulation Stable Diffusion Stable Signature DAG Yu et al. GAN fingerprinting
Closest papers by meaning
- ManiFPT: Defining and Analyzing Fingerprints of Generative Models (CVPR 2024 · similarity 0.87)
- SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models (CVPR 2025 · similarity 0.87)
- OmniMark: Efficient and Scalable Latent Diffusion Model Fingerprinting (AAAI 2025 · similarity 0.86)
- MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning (CVPR 2024 · similarity 0.86)
- Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy (NeurIPS 2024 · similarity 0.86)
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.