Research map / Image generation / Fast diffusion sampling
Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Smoothed Energy Guidance (SEG), a training- and condition-free guidance for diffusion models that blurs self-attention weights (implemented efficiently via query blurring) to produce an unconditional prediction, applied to SDXL.
Problem
Classifier-free guidance requires conditioning, so it cannot be applied to unconditional diffusion models; existing heuristic unconditional guidance methods (SAG, PAG) cause side effects such as smoothing, saturation, color shifts and structural changes at large guidance scales.
Approach
Define an energy for self-attention weights (log-sum-exp based), show Gaussian blurring of attention weights reduces the energy-landscape curvature, use the blurred-attention prediction as the negative term in guidance, and control its strength via the Gaussian sigma instead of the guidance scale; query blurring gives the same effect without quadratic cost.
Key results
- Unconditional SDXL: SEG (sigma->inf) reaches FID 88.215 vs 129.496 for vanilla SDXL, 105.271 for PAG, with LPIPS_vgg 0.536 vs PAG 0.542 (lower side effects).
- Text-conditional SEG sweep: FID drops from 53.423 (vanilla) to 26.169 at sigma->inf, and CLIP score rises from 0.271 to 0.292.
- Increasing SEG guidance scale does not reliably improve FID/CLIP, while increasing sigma does, supporting sigma as the quality control knob.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SEG with σ = 10 or σ → ∞ instead of vanilla guidance | SDXL unconditional generation · Unconditional generation | FID: 129.496 -> 95.316; 129.496 -> 88.215 | Helped |
| Apply SEG with σ = 10 instead of vanilla guidance | SDXL unconditional generation · Unconditional generation | LPIPSvgg: 0.706 -> 0.522; 0.542 -> 0.522 | Helped |
| Increase σ for SEG (ablation) | SDXL text-conditional generation · Text-conditional generation | FID: 53.423 -> 48.284 -> 41.784 -> 33.819 -> 29.325 -> 26.169 | Helped |
| Increase σ for SEG (ablation) | SDXL text-conditional generation · Text-conditional generation | CLIP Score: 0.271 -> 0.273 -> 0.278 -> 0.285 -> 0.290 -> 0.292 | Helped |
| Increase σ for SEG (ablation) | SDXL text-conditional generation · Text-conditional generation | LPIPSvgg: 0.361 -> 0.410 -> 0.449 -> 0.472 -> 0.493 | Hurt |
| Increase γseg instead of controlling generation with σ (ablation) | SEG · Image generation; tested on 30k samples per combination | FID | No clear effect |
| Use key blurring instead of query blurring (ablation) | SEG · Image generation | image quality | No clear effect |
Surprising findings
- Increasing the guidance scale gamma_seg does not generally improve quality due to saturation, whereas increasing sigma does.
- Even the extreme sigma-to-infinity case (attention set to uniform) yields high-quality outputs.
Limitations
- Quality is bounded by the baseline model used (SDXL).
- Application to temporal attention in video or multi-view diffusion models is not addressed.
- Proofs rely on approximations and assumptions (e.g., small softmax values, padding-based statistics).
Experimental setup
Datasets: MS-COCO 2014 validation set
Benchmarks: MS-COCO 2014 validation (30k references)
Metrics: FID CLIP score LPIPS (vgg) LPIPS (alex)
Baselines: Vanilla SDXL SAG PAG
Builds on: classifier-free guidance self-attention guidance perturbed attention guidance modern Hopfield network
Closest papers by meaning
- REG: Rectified Gradient Guidance for Conditional Diffusion Models (ICML 2025 · similarity 0.88)
- Guiding a Diffusion Model with a Bad Version of Itself (NeurIPS 2024 · similarity 0.87)
- Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Feedback Guidance of Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Manifold Preserving Guided Diffusion (ICLR 2024 · similarity 0.86)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.