Research map / Image generation / Text-to-image diffusion
Fast Data Attribution for Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills the slow unlearning-based attribution method AbU+ into a learned DINO+CLIP-Text embedding trained with pointwise cross-entropy learning-to-rank, enabling millisecond-latency data attribution for text-to-image models including Stable Diffusion.
Problem
Data attribution for text-to-image models is too computationally expensive per query (hours for unlearning-based methods, tens of seconds for influence methods) to be practical, especially for large models like Stable Diffusion trained on LAION.
Approach
Use a two-stage curation (off-the-shelf kNN retrieval, then AbU+ reranking) to collect attribution ranks, then learn a small MLP on top of frozen DINO+CLIP-Text features with a pointwise binary cross-entropy ranking loss so attribution reduces to cosine similarity search over precomputed embeddings.
Key results
- On MSCOCO, the tuned DINO+CLIP-Text embedding reaches 18.7 ms latency and 354 MB storage, with loss deviation 9.35e-3 at k=4000 versus 10.70e-3 for AbU+ (2.28 hr) and 9.59e-3 for D-TRAK (46.7 s).
- Among fast methods (faster than generation), the learned embedding performs best on all counterfactual metrics on MSCOCO, as reported in Table 1.
- The method is reported to be 2,500x to 400,000x faster than existing methods while achieving better or competitive attribution performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill AbU+ ranks into a tuned DINO+CLIP-Text embedding | MSCOCO attribution · MSCOCO leave-500-out | MSE image deviation: 5.64 ±0.25 -> 4.78 ±0.22 | Hurt |
| Distill AbU+ ranks into a tuned DINO+CLIP-Text embedding | MSCOCO attribution · MSCOCO leave-500-out | Loss deviation: 5.83 ±0.17 -> 5.28 ±0.17 | Hurt |
| Distill AbU+ ranks into a tuned DINO+CLIP-Text embedding | MSCOCO attribution · MSCOCO leave-500-out | CLIP similarity: 7.15 ±0.10 -> 7.37 ±0.09 | Hurt |
| Replace diagonal Fisher approximation with EK-FAC (ablation) | AbU · MSCOCO attribution | Loss deviation | Helped |
| Add random non-neighbor examples with probability 0.1 (ablation) | Attribution ranker · MSCOCO rank prediction | Accuracy: 0.709 to 0.724 | Helped |
| Increase the non-neighbor sampling rate beyond 0.1 (ablation) | Attribution ranker · MSCOCO rank prediction | Accuracy | Hurt |
| Use cross-entropy rank loss instead of MSE rank regression (ablation) | Attribution ranker · MSCOCO rank prediction | mAP | Helped |
| Use cross-entropy rank loss instead of ordinal loss (ablation) | Attribution ranker · MSCOCO rank prediction | mAP | No clear effect |
| Predict absolute attribution scores instead of ranks (ablation) | Attribution ranker · MSCOCO rank prediction | mAP (1000): 0.724 -> 0.009 | Hurt |
| Restrict AbU+ scoring to top-neighbor candidates using K-NN retrieval (ablation) | AbU+ · MSCOCO leave-K-out | Loss deviation: 5.8 ±0.2 -> 5.8 ±0.2; 7.1 ±0.2 -> 7.1 ±0.2 | No clear effect |
| Combine DINO image features with CLIP-Text features (ablation) | Attribution ranker · MSCOCO rank prediction | mAP | Helped |
| Tune the combined DINO+CLIP-Text feature for attribution (ablation) | Attribution ranker · Stable Diffusion rank prediction | mAP | Helped |
Surprising findings
- On Stable Diffusion, text features are strictly necessary for good tuned performance; DINO and CLIP-image features alone significantly underperform, suggesting AbU+ attributions correlate more with text similarity.
- Pointwise MSE regression of dense ranks fails to converge, while a simple cross-entropy on normalized ranks matches the ordinal loss.
Limitations
- Distills only relative rankings, losing the degree and diffuseness of influence.
- Failure modes of the teacher AbU+ are inherited by the distilled model.
- Ground-truth validity of attributions on Stable Diffusion cannot be verified by counterfactual retraining.
Experimental setup
Datasets: MSCOCO LAION-400M DiffusionDB
Benchmarks: MSCOCO counterfactual leave-K-out test (AbU 110-query test set) LAION-400M rank prediction (Stable Diffusion)
Metrics: mAP(L) loss deviation MSE generation deviation CLIP generation deviation latency storage
Baselines: AbU AbU+ TRAK JourneyTRAK D-TRAK DINO DINOv2 CLIP AbC
Builds on: Attribution by Unlearning (AbU) EK-FAC TRAK D-TRAK
Closest papers by meaning
- Data Attribution for Text-to-Image Models by Unlearning Synthesized Images (NeurIPS 2024 · similarity 0.92)
- Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models (ICLR 2025 · similarity 0.88)
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.87)
- Intriguing Properties of Data Attribution on Diffusion Models (ICLR 2024 · similarity 0.87)
- quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.87)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.