Research map / Image generation / Text-to-image diffusion
Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes inference-time mitigation of diffusion memorization by adjusting the initial Gaussian noise (batch-wise and per-sample sharpness-based updates) so the denoising trajectory escapes the attraction basin earlier, allowing CFG to be applied sooner.
Problem
Text-to-image diffusion models memorize and replicate training images; prior attraction-basin deferral of CFG avoids memorization but yields poorly prompt-aligned images when CFG is applied too late.
Approach
The initial noise sample determines when the trajectory escapes the attraction basin; adjusting x_T via a sharpness-based update that reduces the magnitude of the conditional noise prediction shifts the transition point earlier. Batch-wise adjusts a batch of noises collectively with a few NFEs; Per-sample backpropagates per noise until the transition point disappears.
Key results
- Per-sample mitigation gives the most favorable SSCD-CLIP trade-off across Stable Diffusion v1.4 and v2.0 versus the listed baselines.
- Batch-wise adjustment achieves CLIP-SSCD trade-offs comparable to cross-attention scaling on SD v1.4 and the best performance on SD v2.0.
- Adjusted initial noise yields lower SSCD than unadjusted cases at comparable CLIP scores even when CFG is applied earlier (higher tau).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adjust initial noise to reduce the conditional-noise magnitude (Batch-wise) | Stable Diffusion v1.4 and v2.0 · Memorized prompts | SSCD | Helped |
| Adjust each initial noise sample by minimizing conditional-noise magnitude (Per-sample) | Stable Diffusion v1.4 and v2.0 · Memorized prompts | SSCD–CLIP trade-off | Helped |
| Use Batch-wise initial-noise adjustment | Stable Diffusion v1.4 · Memorized prompts | SSCD–CLIP trade-off | Mixed |
| Increase adjustment strength | Initial-noise adjustment · Memorized prompts; denoising without CFG | Transition point | Helped |
| Increase weight decay from w=0.01 to w=0.2 (ablation) | Per-sample mitigation · Stable Diffusion v1.4; l_target=0.9 | SSCD: 0.2265 -> 0.2275 | No clear effect |
| Increase weight decay from w=0.01 to w=0.2 (ablation) | Per-sample mitigation · Stable Diffusion v1.4; l_target=0.9 | CLIP: 0.2647 -> 0.2657 | No clear effect |
| Increase weight decay from w=0.01 to w=0.2 (ablation) | Per-sample mitigation · Stable Diffusion v1.4; l_target=0.9 | LPIPS: 0.7615 -> 0.7610 | No clear effect |
| Increase weight decay from w=0.01 to w=0.2 (ablation) | Per-sample mitigation · Stable Diffusion v1.4; l_target=0.9 | ImageReward: -0.3522 -> -0.3501 | No clear effect |
| Increase weight decay (ablation) | Adjusted initial noise · Initial noise distributions | JSD: 0.0665 -> 0.046 | Helped |
Surprising findings
- Initial noise choice, long assumed to have minimal effect on memorization mitigation, strongly shifts the transition point where the attraction basin is escaped.
- Per-sample adjustment can eliminate the transition point entirely, allowing CFG from the first timestep without memorization.
Limitations
- Per-sample mitigation adds computational overhead in memory and inference time.
- Batch-wise performance is sensitive to hyperparameter choice and varies across prompts and initial samples.
- Evaluated only on Stable Diffusion v1.4 and v2.0 with memorized prompts from prior work.
Experimental setup
Datasets: LAION (memorized prompts)
Benchmarks: memorized prompts from LAION for SD v1.4 (500 prompts) memorized prompts for SD v2.0 (219 prompts)
Metrics: SSCD CLIP score LPIPS ImageReward
Baselines: Random Token Addition Prompt Embedding Adjustment Cross-attention Scaling Opposite Guidance with Dynamic Transition Point Static Transition Point (CFG deferral) No Mitigation
Builds on: attraction basin (opposite guidance / CFG deferral) Sharpness-Aware Minimization DDIM Stable Diffusion
Closest papers by meaning
- Classifier-Free Guidance Inside the Attraction Basin May Cause Memorization (CVPR 2025 · similarity 0.94)
- Towards Memorization-Free Diffusion Models (CVPR 2024 · similarity 0.92)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.92)
- Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models (AAAI 2025 · similarity 0.90)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.90)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.