Research map / Image generation / Text-to-image diffusion
Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineXtract, which extrapolates between pretrained and fine-tuned diffusion model noise predictions (model guidance) and clusters generated samples to extract fine-tuning training images from personalized DMs.
Problem
Extracting fine-tuning training data (leakage and copyright evidence) from few-shot personalized diffusion model checkpoints shared publicly online, where prior extraction work targets general memorization rather than fine-tuning data.
Approach
Approximate the fine-tuned model's distribution as a geometric interpolation between the pretrained and fine-tuned data distributions, so extrapolating the pretrained and fine-tuned denoising predictions ("model guidance") steers sampling toward high-density regions of the fine-tuning data; generated samples are then clustered via an SSCD similarity graph to pick the most probable training images.
Key results
- On WikiArt with DreamBooth, FineXtract reaches AS 0.449 and A-ESR@0.6 0.22 versus 0.396 and 0.11 for CFG+Clustering.
- On real-world HuggingFace checkpoints, FineXtract raises AS by at least 0.03 and roughly doubles A-ESR in most cases, extracting about 20% of fine-tuning data in most cases.
- Real-world DreamBooth checkpoint results: AS 0.533 and A-ESR@0.6 0.38 for FineXtract versus 0.468 and 0.20 for CFG+Clustering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply model guidance instead of CFG+Clustering | FineXtract extraction · WikiArt, DreamBooth fine-tuning | AS: 0.396 -> 0.449 | Helped |
| Apply model guidance instead of CFG+Clustering | FineXtract extraction · DreamBooth dataset, DreamBooth fine-tuning | A-ESR₀.₇: 0.15 -> 0.25 | Helped |
| Use model guidance rather than CFG alone | FineXtract · WikiArt, SD (V1.4) | AS: 0.434 -> 0.501 | Helped |
| Increase the number of generated images (ablation) | FineXtract · WikiArt, varying generated-image count | extraction performance | Helped |
| Increase the number of training images (ablation) | FineXtract · WikiArt, varying training-image count | extraction performance | Hurt |
| Set guidance scale w' to 3.0 (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | AS | Helped |
| Set correction term scale k to -0.02 (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | extraction performance | Helped |
| Train with Cutout defense (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | AS: 0.501 -> 0.397 | Hurt |
| Train with RandAugment defense (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | AS: 0.501 -> 0.267 | Hurt |
| Use extracted caption instead of full caption (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | AS: 0.501 -> 0.314 | Hurt |
| Use empty caption instead of full caption (ablation) | FineXtract · WikiArt, DreamBooth fine-tuning | AS: 0.501 -> 0.192 | Hurt |
| Use model guidance rather than CFG+Clustering | FineXtract · Mixed DreamBooth and WikiArt data | AS: 0.457 -> 0.480 | Helped |
Surprising findings
- Caption extraction via linear projection layers recovers artist names from the fine-tuned checkpoint without access to the training caption.
- Cutout and RandAugment defenses reduce extraction success but render generated images largely unusable.
Limitations
- null
Experimental setup
Datasets: WikiArt DreamBooth
Benchmarks: WikiArt DreamBooth
Metrics: AS A-ESR@0.6 A-ESR@0.7 DINO CLIP-IQA
Baselines: Direct Text2img+Clustering CFG+Clustering
Builds on: classifier-free guidance DreamBooth LoRA Carlini et al. diffusion data extraction
Closest papers by meaning
- CDI: Copyrighted Data Identification in Diffusion Models (CVPR 2025 · similarity 0.88)
- The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline (ICML 2024 · similarity 0.88)
- DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models (ICLR 2024 · similarity 0.88)
- Slight Corruption in Pre-training Data Makes Better Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning (NeurIPS 2024 · similarity 0.87)
- CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.