Research map / Image generation / Text-to-image diffusion
Training-Free Diffusion Model Alignment with Sampling Demons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Demon, a training-free, backpropagation-free inference-time method that steers the denoising SDE by reweighting Gaussian noise candidates scored by a (possibly non-differentiable) reward estimated via PF-ODE or consistency models.
Problem
Aligning pretrained diffusion models with user preferences without retraining or differentiable reward functions, supporting non-differentiable rewards such as VLM APIs and human judgments.
Approach
At each denoising step, sample K Gaussian noises, score each resulting next state by the reward of its PF-ODE clean-image estimate, and combine the noises with tanh (Tanh Demon) or softmax (Boltzmann Demon) weights to synthesize a noise that biases the SDE step toward higher reward; a theory section gives an improvement guarantee under assumptions.
Key results
- Tanh Demon with 5-minute budget on SD v1.4 reaches aesthetics 6.72 ± 0.26 (Tanh) and Tanh-C reaches 7.27 ± 0.33, versus Best-of-N upper bound around 6.5.
- Tanh + Aes reaches 7.35 ± 0.40 aesthetics on the ensemble-comparison table, versus 5.34 for SD v1.4 and 6.32 for Best-of-N.
- Manual user selection raises DINOv2 cosine similarity to the reference cat image from 0.594 (PF-ODE) to 0.708.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tanh Demon with Aes reward instead of unaligned sampling | Stable Diffusion v1.4 · Animal prompts | Aes: 5.34 ± 0.56 -> 7.35 ± 0.40 | Helped |
| Use Tanh Demon with Ensemble reward instead of Best-of-N | Stable Diffusion v1.4 · Animal prompts | Pick: 0.218 ± 0.009 -> 0.236 ± 0.014 | Helped |
| Use online human selections to guide sampling | Diffusion model · Manual reference-image matching | DINOv2 cosine similarity: 0.594 -> 0.708 | Helped |
| Replace the ODE reward estimator with a 1-step consistency model (ablation) | Tanh Demon · Animal prompts, 5-minute generation | Aes: 6.72 ± 0.26 -> 7.27 ± 0.33 | Helped |
| Increase T from 16 to 32 sampling steps (ablation) | Tanh Demon · Animal prompts | Aes: 6.723 ± 0.26 -> 7.073 ± 0.22 | Helped |
| Increase T from 32 to 64 sampling steps (ablation) | Tanh Demon · Animal prompts | Aes: 7.073 ± 0.22 -> 7.394 ± 0.29 | Helped |
| Increase K from 16 to 64 noise samples (ablation) | Tanh Demon · Animal prompts | Aes: 7.549 ± 0.43 -> 8.566 ± 0.33 | Helped |
| Increase beta from 0.05 to 0.1 (ablation) | Tanh Demon · Animal prompts | Aes: 7.459 ± 0.33 -> 7.841 ± 0.32 | Helped |
| Increase K from 8 to 16 noise samples (ablation) | Tanh Demon · Animal prompts | Aes: 7.446 ± 0.37 -> 7.841 ± 0.32 | Helped |
| Increase K from 16 to 32 noise samples (ablation) | Tanh Demon · Animal prompts | Aes: 7.841 ± 0.32 -> 8.179 ± 0.35 | Helped |
| Set temperature tau=1 instead of adaptive tau (ablation) | Tanh Demon · Animal prompts | Aes: 7.45 ± 0.33 -> 7.40 ± 0.30 | No clear effect |
| Set temperature tau=0.01 instead of adaptive tau (ablation) | Tanh Demon · Animal prompts | Aes: 7.45 ± 0.33 -> 7.24 ± 0.31 | Hurt |
Surprising findings
- The 1-step consistency model reward estimate is less accurate than 20-step ODE but is faster and, under limited time budgets, gives better final reward than the 20-step ODE variant.
- Tanh Demon outperforms Boltzmann Demon empirically, despite Boltzmann's theoretical expected-improvement guarantee.
Limitations
- Relies on the reward estimate via PF-ODE being close to the expected SDE reward; theory assumes r_beta equals r composed with c and negligible truncation error.
- Compute cost scales as K*T reward evaluations, so higher quality requires many reward queries and longer sampling time.
- Accuracy of VLM pairwise comparisons degrades with larger candidate sets, so the method uses binary Quicksort-partition comparisons.
Experimental setup
Datasets: ImageNet-1K animal names (45 animals) prompts prompts from prior work
Benchmarks: LAION aesthetics predictor (Aes) on animal prompts ImageReward PickScore HPSv2
Metrics: aesthetics score ImageReward PickScore HPSv2 DINOv2 cosine similarity wall-clock time reward queries
Baselines: Best-of-N DOODL DPO Universal Guidance (Uni) DDPO
Builds on: EDM DOODL DDPO DPO
Closest papers by meaning
- Inference-Time Alignment of Diffusion Models with Direct Noise Optimization (ICML 2025 · similarity 0.91)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.91)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Composition and Alignment of Diffusion Models using Constrained Learning (NeurIPS 2025 · similarity 0.89)
- Stitched Value Model for Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
- Fast constrained sampling in pre-trained diffusion models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.