Research map / Image generation / Text-to-image diffusion
DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Plants element-level injected memorization into text-to-image diffusion models by coating protected images with stealthy warping functions, then detects unauthorized training via a signal classifier and hypothesis testing on generated images.
Problem
Detecting whether a text-to-image diffusion model was trained or fine-tuned on protected images without permission, which membership inference handles poorly for large diffusion models.
Approach
Instead of sample-level memorization, modify protected images with imperceptible image warping (a signal function) so that models trained on them memorize this element; detect misuse by classifying whether generated images contain the signal and testing whether the rate exceeds a threshold, with optional trigger words for conditional memorization.
Key results
- Detection accuracy of 100.0% across fine-tuning settings (Stable Diffusion v1/v2 with LoRA, DreamBooth) with 20 models per class and zero FP/FN.
- Memorization strength averages 91.2% for models with unauthorized usage vs 5.1% without in fine-tuning; 98.0% vs 5.0% for VQ Diffusion standard training on CUB-200.
- On a multi-source setting at 25% collect fraction, DIAGNOSIS achieves 100% accuracy while the compared existing method achieves 50.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Plant unconditional or trigger-conditioned injected memorization in protected data | DIAGNOSIS · Fine-tuning on Pokemon, CelebA, CUB-200, and Dog | detection accuracy: 100.0% | Helped |
| Use DIAGNOSIS instead of the comparison method [cite] | Stable Diffusion v1 + LoRA · Pokemon; multiple data sources, collect fraction 25% | detection accuracy: 50.0% -> 100.0% | Helped |
| Add unconditional injected memorization | Stable Diffusion v1 + LoRA · Pokemon | FID: 199.29 -> 218.28 | Hurt |
| Add trigger-conditioned memorization and include the text trigger | Stable Diffusion v1 + LoRA · Pokemon; triggered prompts | FID: 199.29 -> 239.03 | Hurt |
| Increase warping strength from 1.0 to 4.0 (ablation) | DIAGNOSIS, unconditional · Pokemon; Stable Diffusion v1 + LoRA | memorization strength: 76.0% -> 100.0% | Helped |
| Increase warping strength from 1.0 to 4.0 (ablation) | DIAGNOSIS, unconditional · Pokemon; Stable Diffusion v1 + LoRA | FID: 209.78 -> 262.30 | Hurt |
| Increase unconditional coating rate from 2.0% to 100.0% (ablation) | DIAGNOSIS, unconditional · Pokemon; Stable Diffusion v1 + LoRA | memorization strength: 0.0% -> 96.0% | Helped |
| Vary trigger-conditioned coating rate from 5.0% to 50.0% (ablation) | DIAGNOSIS, trigger-conditioned · Pokemon; Stable Diffusion v1 + LoRA | memorization strength: 100.0% | No clear effect |
| Use word trigger instead of syntactic trigger (ablation) | DIAGNOSIS, trigger-conditioned · Pokemon; Stable Diffusion v1 + LoRA | detection accuracy: 100.0% | No clear effect |
| Replace image warping with the 1977, Kelvin, or Toaster Instagram filter (ablation) | DIAGNOSIS, unconditional · Five models with unauthorized usage and five without | detection accuracy: 100.0% | No clear effect |
| Add strong training-time augmentations (compression, blur, sharpening, noise, or color jittering) (ablation) | DIAGNOSIS, unconditional · Pokemon; Stable Diffusion v1 + LoRA | detection accuracy: 100.0% | No clear effect |
| Increase coating rate from 5.0% to 50.0% for trigger-conditioned memorization (ablation) | DIAGNOSIS, trigger-conditioned · Pokemon; Stable Diffusion v1 + LoRA | FID: 238.87 -> 247.97 | Mixed |
Surprising findings
- Trigger-conditioned memorization remains detectable at coating rates as low as 2%, while unconditional memorization fails at low coating rates.
Limitations
- Unconditional memorization requires a relatively high coating rate and fails to detect malicious models at very low coating rates.
- Trigger-conditioned memorization is only suitable for Scenario I where the infringer uses captions from the protector.
- Strong augmentations raise FID substantially, though detection stays at 100% in the adaptive-infringer tests.
Experimental setup
Datasets: Pokemon CelebA CUB-200 Dog Cat-Dog-Bird
Benchmarks: Pokemon CelebA CUB-200 Dog
Metrics: detection accuracy TP/FP/FN/TN FID memorization strength SSIM PSNR MAE MSE
Baselines: Radioactive data-style detection (Sablayrolles et al.) SecMI Glaze
Builds on: Stable Diffusion LoRA DreamBooth Textual Inversion Glaze
Closest papers by meaning
- Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy (NeurIPS 2024 · similarity 0.90)
- Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models (ICML 2025 · similarity 0.88)
- The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models (CVPR 2025 · similarity 0.86)
- CDI: Copyrighted Data Identification in Diffusion Models (CVPR 2025 · similarity 0.86)
- The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.