Research map / Image generation / Fast diffusion sampling
A General Framework for Inference-time Scaling and Steering of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Feynman-Kac (FK) steering, an inference-time particle-resampling framework that steers text-to-image and text diffusion models with arbitrary (including non-differentiable) reward functions, without any training.
Problem
Steering diffusion models toward user-specified properties (prompt fidelity, aesthetics, text quality, toxicity) is costly with fine-tuning and limited for best-of-n or gradient guidance, which need differentiable rewards and continuous states.
Approach
Treats sampling from the reward-tilted distribution as a Feynman-Kac interacting particle system: k diffusion particles are resampled at intermediate steps using potentials built from intermediate rewards (evaluated on denoised estimates, many-sample averages, or learned reward models), generalizing TDS and SVDD and working for discrete and continuous diffusion.
Key results
- On GenEval, FK steering with SDv2.1 (0.87B, k=4) reaches 0.629 versus 0.581 for SDXL-DPO (2.6B fine-tuned) and 0.557 for base SDXL, with faster sampling (11.5s vs 9.1s per the paper's text).
- For SSD-LM toxicity, FK with k=8 and learned rewards raises the toxic fraction from 0.4% to 39.0%, above gradient guidance at 22.3%.
- For MDLM, FK with k=8 and many-sample rewards raises toxicity to 64.7% from 0.3% base, beating best-of-8 at 3.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply FK steering with 4 particles | Stable Diffusion text-to-image models · GenEval | GenEval score: SD v1.4: 0.4408 -> 0.5492; SD v1.5: 0.4483 -> 0.5463; SD v2.1: 0.5104 -> 0.6289; SDXL: 0.5571 -> 0.6489 | Helped |
| apply FK steering to the fine-tuned model | SD v1.5-DPO and SDXL-DPO · GenEval | GenEval score: 0.4671 -> 0.5751; 0.5811 -> 0.6755 | Helped |
| use FK with gradient-guided proposal | SD v1.5 · GenEval | GenEval score: 0.54 -> 0.56 | Helped |
| use GPT2 reward for FK steering | SSD-LM · Text quality | PPL (GPT-XL): 23.2 -> 11.0 | Helped |
| use CoLA reward for FK steering | SSD-LM · Text quality | CoLA: 68.3 -> 95.7 | Helped |
| use trigram reward for FK steering | MDLM · Text quality | PPL (GPT-XL): 85.3 -> 40.3 | Helped |
| use CoLA reward for FK steering | MDLM · Text quality | CoLA: 28.9 -> 69.8 | Helped |
| increase particle count from 4 to 8 (ablation) | FK steering · GenEval and human preference alignment | score | Helped |
| use max rather than sum or difference potential (ablation) | FK text-to-image steering · GenEval | GenEval score: SD v1.4, k=4: 0.496 -> 0.540; 0.525 -> 0.540 | Helped |
| use learned intermediate reward (ablation) | FK ImageNet class-conditional generation · ImageNet class conditioning | p(y | x0) mean: 0.59 -> 0.88 | Helped |
| use learned intermediate reward (ablation) | FK steering of SSD-LM · Text toxicity | Toxic (Holdout): 29.7% -> 38.0% | Helped |
| increase intermediate reward samples from 4 to 16 (ablation) | FK steering of MDLM · Text toxicity | Toxic (Holdout): 48.3% -> 51.7% | Helped |
Surprising findings
- Best-of-4 alone already outperforms DPO and DDPO fine-tuned models on prompt fidelity and aesthetic scores.
- The max potential gives higher prompt fidelity but lower particle diversity than the difference potential.
Limitations
- Performance depends on availability of strong reward functions.
- Sampling cost scales with particle count and reward evaluations, and learned rewards add training or extra evaluation cost.
- Max potential can reduce sample diversity.
Experimental setup
Datasets: GenEval prompts ImageReward prompts OpenWebText CoLA ImageNet
Benchmarks: GenEval ImageReward benchmark prompts HPS GPT2-XL perplexity CoLA toxicity classifier rates
Metrics: GenEval score ImageReward HPS perplexity CoLA accuracy Dist-1/2/3 toxic fraction CLIP diversity
Baselines: best-of-n sampling DPO fine-tuned SD v1.5 and SDXL DDPO fine-tuned SD v1.4 gradient guidance SSD-LM with T=5000 GPT2-medium
Builds on: sequential monte carlo twisted diffusion sampler (TDS) SVDD classifier guidance DDIM
Closest papers by meaning
- Diffusion Tree Sampling: Scalable inference‑time alignment of diffusion models (NeurIPS 2025 · similarity 0.90)
- Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.90)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025 · similarity 0.88)
- A Mean-Field Framework for Inference-Time Distributional Control of Diffusion Models (NeurIPS 2026 · similarity 0.88)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.