Research map / Image generation / Text-to-image diffusion

Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models

NeurIPS 2025 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HeadHunter, an iterative per-attention-head selection framework for perturbation guidance in DiT diffusion/flow models, plus SoftPAG, which linearly interpolates selected heads' attention maps toward the identity to control perturbation strength.

Problem

Layer-level attention perturbation guidance (e.g., PAG) relies on heuristic layer choice and oversmooths outputs in DiT architectures, where quality-relevant computation is spread across layers; the paper asks where within attention perturbations should be applied.

Approach

Perturb individual attention heads rather than whole layers, select heads greedily over rounds using a user-defined objective (e.g., PickScore, aesthetic score), and continuously control strength by interpolating each selected head's attention map toward the identity matrix (SoftPAG).

Key results

• Using only 6 of 24 PickScore-selected heads in SD3 matches or beats full layer-level PAG on FID, with further gains from more heads.
• For style-oriented quality, HeadHunter with 50 unseen content prompts achieves significantly higher human preference than the baseline and comparable preference to CFG.
• SoftPAG with interpolation u=0.5 yields better FID than full identity replacement (u=1) in the reported grid search, with the best setting often at u<1.

Measured effects reported in the paper

Surprising findings

• Some heads that produce low-quality outputs alone contribute strongly to style when composed with other heads, so one-shot evaluation would miss them.
• Heads selected for general quality concentrate in early-to-mid layers (2.9% in layers >=16), while style heads spread across depth (24.8% in layers >=16).

Limitations

• Head selection requires repeated generation over a candidate pool and prompt-seed sets, which is compute-intensive.
• Excessive numbers of perturbed heads cause oversaturation or oversimplification, and the objective's reward-model bias can induce artifacts.

Experimental setup

Datasets: MS COCO validation (1K prompts for FID)

Benchmarks: MS COCO FID (1K prompts)

Metrics: FID PickScore ImageReward MPS LAION aesthetic score Precision Recall Density Coverage human preference

Baselines: PAG SEG classifier-free guidance Uniform Guidance Max Guidance

Builds on: Perturbed-Attention Guidance (PAG) Smoothed Energy Guidance (SEG) classifier-free guidance flow matching

Closest papers by meaning

• Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models (CVPR 2024 · similarity 0.90)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.89)
• Token Perturbation Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.89)
• Devil is in the Details: Density Guidance for Detail-Aware Generation with Flow Models (ICML 2025 · similarity 0.89)
• Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization (ICLR 2025 · similarity 0.87)
• What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.87)