Research map / Image generation / Text-to-image diffusion
Data Attribution for Text-to-Image Models by Unlearning Synthesized Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes attributing text-to-image outputs by unlearning the synthesized image (EWC-regularized Newton updates on cross-attention K/V) and ranking training images by their loss increase, validated by leave-K-out retraining on MSCOCO and the Customized Model benchmark.
Problem
Identifying which training images most influence a generated image in text-to-image diffusion models; exact counterfactual (leave-K-out retraining) is computationally infeasible, and existing approximations (influence functions, feature matching) are weak for large-scale generative models.
Approach
Instead of unlearning each training image separately, unlearn the synthesized image once with EWC-regularized updates restricted to cross-attention key/value weights, then score each training image by its DDPM loss increase after unlearning.
Key results
- On MSCOCO leave-K-out at K=4000, the method achieves the largest loss change (9.8e-3), tied near the top with D-TRAK on regeneration MSE/CLIP deviation.
- Removing 500 images with the method (0.42% of MSCOCO) yields generation deviation equal to removing ~57.5k random images.
- On the Customized Model benchmark, the method outperforms DINO (AbC), CLIP (AbC), and D-TRAK, especially on artist-style models, with Recall@10 and mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attribution by unlearning the synthesized image | Text-to-image model attribution · MSCOCO leave-K-out, K=500 | ΔL (x10^-3): 5.6 ±0.16 vs. 5.4 ±0.16 | Helped |
| Use attribution by unlearning the synthesized image | Text-to-image model attribution · MSCOCO leave-K-out, K=4000 | ΔL (x10^-3): 9.8 ±0.32 vs. 9.6 ±0.33 | Helped |
| Use attribution by unlearning the synthesized image | Text-to-image model attribution · MSCOCO leave-K-out, K=500 | Generation deviation MSE (x10^-2): 5.1 ±0.21 vs. 5.9 ±0.24 | Hurt |
| Use attribution by unlearning the synthesized image | Text-to-image model attribution · MSCOCO leave-K-out, K=500 | Generation deviation CLIP (x10^-1): 7.3 ±0.09 vs. 7.3 ±0.10 | No clear effect |
| Add Fisher-information EWC regularization during unlearning (ablation) | Synthesized-image unlearning attribution · MSCOCO leave-K-out, K=500 | ΔL (x10^-3): 3.5 ±0.10 -> 5.5 ±0.17 | Helped |
| Add Fisher-information EWC regularization during unlearning (ablation) | Synthesized-image unlearning · Target synthesized images | MSE: 0.081 ±0.003 -> 0.086 ±0.005 | Mixed |
| Update a subset of weights rather than all model weights (ablation) | Synthesized-image unlearning attribution · MSCOCO leave-K-out, K=4000 | ΔL (x10^-3): 8.4 ±0.27 -> 9.8 ±0.32 | Helped |
| Update cross-attention key and value weights (ablation) | Synthesized-image unlearning attribution · MSCOCO leave-K-out, K=500 | ΔL (x10^-3): 5.5 ±0.17 -> 5.6 ±0.16 | Helped |
| Use one SGD step without EWC (ablation) | Synthesized-image unlearning attribution · MSCOCO leave-K-out, K=500 | ΔL (x10^-3): 5.6 ±0.16 -> 3.5 ±0.10 | Hurt |
| Use ten SGD steps without EWC (ablation) | Synthesized-image unlearning attribution · MSCOCO leave-K-out, K=500 | ΔL (x10^-3): 5.6 ±0.16 -> 3.4 ±0.10 | Hurt |
| Update all model weights instead of cross-attention key and value weights (ablation) | Synthesized-image unlearning · Target synthesized images | MSE: 0.093 ±0.004 vs. 0.086 ±0.005 | Hurt |
| Replace the DDPM denoising loss with the square loss used by D-TRAK (ablation) | Our attribution method · MSCOCO leave-K-out | ΔL (x10^-3): 5.6 ±0.16 -> 4.4 ±0.14; 6.7 ±0.22 -> 5.1 ±0.20; 9.8 ±0.32 -> 7.6 ±0.36 | Hurt |
Surprising findings
- Applying D-TRAK's square-loss change to the unlearning-based method reduces performance, unlike its large gain for D-TRAK.
- DataInf performs poorly, close to random, on text-to-image models trained from scratch.
Limitations
- Estimating training losses over the full training set remains a bottleneck requiring many forward passes.
- Scores are per-image and ignore interactions among groups of influential images.
- Attribution covers whole images or regions; finer attributes such as style or segments are not addressed.
Experimental setup
Datasets: MSCOCO 2017 LAION subset Customized Model benchmark
Benchmarks: MSCOCO leave-K-out retraining Customized Model benchmark
Metrics: DDPM loss change MSE deviation of regenerated image CLIP deviation Recall@10 mAP
Baselines: Random Pixel similarity CLIP text DINOv2 CLIP image DINO CLIP (AbC) DINO (AbC) DataInf TRAK JourneyTRAK D-TRAK
Builds on: elastic weight consolidation influence functions TRAK Attribution by Customization JourneyTRAK
Closest papers by meaning
- Fast Data Attribution for Text-to-Image Models (NeurIPS 2025 · similarity 0.92)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.88)
- Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.87)
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.86)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.