Research map / Image generation / Fast diffusion sampling
An Efficient Framework for Crediting Data Contributors of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sparsified fine-tuning (magnitude pruning plus subset fine-tuning) to make Shapley-value estimation of data contributors' credit for global properties of diffusion models computationally feasible.
Problem
Crediting data contributors for global properties (e.g., image quality, demographic diversity, aesthetic quality) of diffusion models requires retraining and re-evaluating models on many contributor subsets, which is computationally impractical for Shapley value estimation.
Approach
Prune the trained diffusion model once, fine-tune the pruned model on each contributor subset for a few steps instead of retraining from scratch, and use KernelSHAP-style weighted least squares over these subsets to estimate Shapley values for contributors.
Key results
- Sparsified-FT Shapley achieves LDS of 61.48% on CIFAR-20, 26.34% on CelebA-HQ, and 61.44% on ArtBench at alpha=0.5, the highest among baselines (best baselines: LOO 30.66%, aesthetic score 24.85%, pixel similarity max 21.70%).
- Retraining plus inference with sparsified fine-tuning takes 18.3, 22.9, and 10.5 minutes per Shapley subset, which is 5.3, 10.4, and 18.6 times faster than retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparsified-FT Shapley instead of LOO | DDPM contributor attribution · CIFAR-20 | LDS (%): 30.66 $\pm$ 6.11 -> 61.48 $\pm$ 2.27 | Helped |
| Use sparsified-FT Shapley instead of maximum pixel similarity | LDM contributor attribution · CelebA-HQ | LDS (%): 21.70 $\pm$ 2.05 -> 26.34 $\pm$ 3.42 | Helped |
| Use sparsified-FT Shapley instead of average aesthetic score | Stable Diffusion LoRA contributor attribution · ArtBench (Post-Impressionism) | LDS (%): 24.85 $\pm$ 2.30 -> 61.44 $\pm$ 2.04 | Helped |
| Add more sparsified fine-tuning steps (ablation) | Sparsified-FT model approximation · 100 sampled contributor subsets | Pearson correlation with retrained model properties | Helped |
| Add more sparsified fine-tuning steps (ablation) | Sparsified-FT Shapley estimates · 500 sampled contributor subsets | Pearson correlation with retrained Shapley values | Helped |
| Increase the number of sparsified fine-tuning steps (ablation) | Sparsified-FT Shapley estimation · Shapley kernel | runtime | Hurt |
| Use sparsified fine-tuning instead of retraining from scratch | Shapley subset model evaluation · CIFAR-20 | speedup: 5.3 times faster | Helped |
| Use sparsified fine-tuning instead of retraining from scratch | Shapley subset model evaluation · CelebA-HQ | speedup: 10.4 times faster | Helped |
| Use sparsified fine-tuning instead of retraining from scratch | Shapley subset model evaluation · ArtBench (Post-Impressionism) | speedup: 18.6 times faster | Helped |
Surprising findings
- TRAK-based methods sometimes yield poor or negative LDS, and Journey-TRAK gives negative LDS for Inception Score and aesthetic score.
- Simple similarity heuristics (e.g., raw pixel similarity, CLIP similarity) occasionally outperform TRAK-based methods.
Limitations
- Theoretical bounds are asymptotic in fine-tuning steps and assume convex quadratic-like objectives; finite-step and Shapley estimation bounds are left for future work.
- Low LDS on CelebA-HQ (26.34%) compared with other datasets.
- Requires access to training data and model parameters.
Experimental setup
Datasets: CIFAR-20 CelebA-HQ ArtBench-10 (Post-Impressionism)
Benchmarks: CIFAR-20 CelebA-HQ ArtBench (Post-Impressionism)
Metrics: linear datamodeling score (Spearman) relative change in global property (counterfactual) Inception Score demographic entropy aesthetic score Pearson correlation runtime
Baselines: Pixel similarity Embedding distance CLIP similarity Gradient similarity Aesthetic score Relative IF Renormalized IF TRAK Journey-TRAK D-TRAK Leave-one-out
Builds on: TRAK D-TRAK Journey-TRAK KernelSHAP lottery ticket hypothesis
Closest papers by meaning
- Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models (ICLR 2025 · similarity 0.90)
- Intriguing Properties of Data Attribution on Diffusion Models (ICLR 2024 · similarity 0.88)
- Training Data Attribution in Diffusion Models via Mirrored Unlearning and Noise-Consistent Skew (NeurIPS 2026 · similarity 0.87)
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.87)
- Balancing Act: Distribution-Guided Debiasing in Diffusion Models (CVPR 2024 · similarity 0.86)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.