Research map / Image generation / Text-to-image diffusion
Exploring Local Memorization in Diffusion Models via Bright Ending Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a "bright ending" cross-attention anomaly in Stable Diffusion that localizes memorized image regions in one inference pass, and uses it to improve local memorization detection, mitigation, and evaluation.
Problem
Existing memorization evaluation, detection, and mitigation methods in diffusion models take a global view and miss local memorization, where only parts of a training image are replicated; they also cannot localize the memorized regions.
Approach
Memorized image patches receive abnormally high cross-attention from the final text (end) token at the last inference step ("bright ending"), so the end-token attention map serves as a training-data-free local memorization mask that is plugged into magnitude-based detection, prompt-optimization mitigation, and a masked L2 similarity metric.
Key results
- Local-memorization detection with BE-masked magnitude (LD) at T=1 improves AUC from 0.918 to 0.943 and T@1%F from 0.629 to 0.731 over the baseline D.
- At T=50, local-memorization detection T@1%F rises from 0.560 (baseline) to 0.926 (LD), while global-memorization performance stays essentially unchanged.
- The BE-based mitigation reduces SD's similarity score for local memorization about twice as much as the baseline while lowering CLIP score by less than half of the baseline's reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Local memorization, T=1/10/50 | AUC: 0.918 -> 0.943; 0.989 -> 0.995; 0.990 -> 0.996 | Helped |
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Local memorization, T=1/10/50 | F1: 0.864 -> 0.893; 0.982 -> 0.987; 0.983 -> 0.988 | Helped |
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Local memorization, T=1/10/50 | T@1%F: 0.629 -> 0.731; 0.953 -> 0.985; 0.560 -> 0.926 | Helped |
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Global memorization, T=1/10/50 | AUC: 0.979 -> 0.981; 1.000 -> 1.000; 0.999 -> 0.999 | No clear effect |
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Global memorization, T=1/10/50 | F1: 0.944 -> 0.948; 0.987 -> 0.987; 0.976 -> 0.977 | Mixed |
| Add the BE mask to the detection magnitude (D to LD) (ablation) | Baseline detection strategy · Global memorization, T=1/10/50 | T@1%F: 0.934 -> 0.940; 1.000 -> 1.000; 1.000 -> 1.000 | No clear effect |
Surprising findings
- Existing global similarity metrics (SSCD >0.5) misclassify locally memorized generations as non-memorized, producing many false negatives.
- Prior evaluation labels memorization per prompt rather than per generation, which can inflate detection metrics.
Limitations
- Requires a full inference pass to the final step, adding latency to detection and mitigation.
- Evaluation of the extracted mask is extrinsic only; no intrinsic comparison against ground-truth masks.
- Manual labeling of ground truth limits scale and may carry subjectivity.
Experimental setup
Datasets: LAION prompts (non-memorized) Lexica prompts (non-memorized) COCO captions (non-memorized) Random captions (non-memorized) Memorized prompt set from prior work (300 selected)
Benchmarks: Local vs global memorization detection on SD v1-4 generations (9,600 images)
Metrics: AUC F1 T@1%F CLIP score SSCD similarity F1 for memorization classification
Baselines: Magnitude-based detection (D) Prompt-embedding mitigation (Wen et al.) SSCD Cross-attention entropy detection Nearest-neighbor global similarity
Builds on: magnitude-based memorization detection SSCD classifier-free guidance
Closest papers by meaning
- Detecting, Explaining, and Mitigating Memorization in Diffusion Models (ICLR 2024 · similarity 0.92)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.91)
- Towards Memorization-Free Diffusion Models (CVPR 2024 · similarity 0.90)
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.