Research map / Image generation / Fast diffusion sampling
Token Perturbation Guidance for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Perturbation Guidance (TPG), a training-free guidance method that shuffles intermediate denoiser tokens to build a negative score, improving unconditional and conditional diffusion sampling on SDXL, SD 2.1 and SD3.
Problem
CFG requires special conditional training and is limited to conditional generation, while existing training-free perturbation guidance methods (PAG, SEG) give limited gains in quality and prompt alignment.
Approach
Apply a norm-preserving, linear token shuffling permutation to intermediate denoiser token representations in a second forward pass to form the negative score; this yields guidance whose direction and frequency profile closely resemble CFG, without training or architecture changes.
Key results
- Unconditional SDXL: TPG FID 69.31 vs vanilla SDXL 124.04 (about 2x improvement) and SEG 82.64.
- Conditional SDXL: CFG FID 12.79 and CLIP 32.03; TPG FID 17.77 and CLIP 30.15, versus SEG 23.94 and PAG 20.49 FID.
- Unconditional SD 2.1: TPG FID 16.69 vs vanilla 25.24, with Inception Score 36.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-shuffling guidance | SDXL · Unconditional generation | FID: 82.64 -> 69.31 | Helped |
| Add token-shuffling guidance | SDXL · Conditional generation | CLIP Score: 32.03 -> 30.15 | Mixed |
| Add token-shuffling guidance | Stable Diffusion 2.1 · Unconditional generation | FID: 20.98 -> 16.69 | Helped |
| Replace token shuffling with sign flipping (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 78.43 -> 119.23 | Hurt |
| Replace token shuffling with Hadamard transform (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 78.43 -> 120.54 | Hurt |
| Replace token shuffling with Haar transform (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 78.43 -> 118.47 | Hurt |
| Increase guidance scale from 0 to 3 (ablation) | TPG on SDXL · Unconditional generation | FID: 136.01 -> 76.00 | Helped |
| Increase guidance scale beyond 3 (ablation) | TPG on SDXL · Unconditional generation | FID: 76.00 -> 77.62 (σ=4); 76.00 -> 84.17 (σ=7) | Hurt |
| Use downsampling layers rather than mid layers for perturbation (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 100.80 -> 76.00 | Helped |
| Use upsampling layers rather than downsampling layers for perturbation (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 102.28 -> 76.00 | Helped |
| Replace token shuffling with token blurring (ablation) | TPG on SDXL · Unconditional generation, 5K samples | FID: 76.00 -> 157.67 | Hurt |
| Add token shuffling instead of using the vanilla model (ablation) | SDXL · Unconditional generation, 5K samples | Inception Score: 7.48 -> 18.47 | Helped |
Surprising findings
- PAG performs worse than vanilla SD3 on DiT-based models, while TPG with shuffle-and-unshuffle works on SD3.
- Non-norm-preserving token blurring degrades all metrics relative to vanilla, whereas norm-preserving shuffling improves them.
Limitations
- Requires two forward passes per denoising step, increasing sampling time versus unguided sampling.
- Guidance may remain limited in extreme out-of-distribution scenarios.
- Conditional CLIP score and FID remain behind CFG in conditional generation.
Experimental setup
Datasets: MS-COCO 2014 validation set
Benchmarks: MS-COCO 2014 validation
Metrics: FID sFID Inception Score CLIP Score Aesthetic Score Precision Recall cosine similarity to noise
Baselines: Vanilla SDXL PAG SEG CFG Vanilla SD 2.1 Vanilla SD3
Builds on: classifier-free guidance Perturbed Attention Guidance Smoothed Energy Guidance Stable Diffusion XL
Closest papers by meaning
- No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models (ICLR 2025 · similarity 0.93)
- Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.92)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models (ICLR 2025 · similarity 0.91)
- Tortoise and Hare Guidance: Accelerating Diffusion Model Inference with Multirate Integration (NeurIPS 2025 · similarity 0.90)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.