Research map / Image generation / Fast diffusion sampling

Test-time Alignment of Diffusion Models without Reward Over-optimization

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DAS, a training-free test-time SMC sampler with tempered intermediate targets that samples from the KL-regularized reward-aligned distribution of diffusion models, matching fine-tuning reward gains while keeping diversity and cross-reward generalization.

Problem

Aligning pretrained diffusion models with reward objectives suffers from reward over-optimization and loss of diversity under fine-tuning, while approximate guidance methods fail to optimize target rewards effectively.

Approach

Sample from the reward-tilted target p_data * exp(r/alpha) using sequential Monte Carlo with a tempered sequence of intermediate targets (likelihood exponent lambda_t increasing from 0 to 1 as t goes T to 0) and a locally optimal proposal approximated via Tweedie's formula, avoiding any model training.

Key results

• On the single-reward experiments, the method reaches target reward comparable to or better than fine-tuning methods (DDPO, AlignProp) while keeping diversity and cross-reward generalization close to guidance methods.
• In online black-box optimization, DAS-UCB reaches unseen-reward ImageReward 1.14 and LPIPS diversity 0.68, versus 0.25/-0.31 ImageReward and 0.51 LPIPS for SEIKO-UCB.
• On the toy mixture-of-Gaussians example, approximate guidance methods are inexact and fine-tuning fits only a single mode, whereas the SMC sampler recovers the target distribution.

Measured effects reported in the paper

Surprising findings

• Multi-reward weight interpolation of separately trained LoRA models is reported to be not Pareto optimal compared to training directly on the weighted reward average.

Limitations

• Sample efficiency depends on number of particles; with fewer particles, manifold deviation rises and KL penalty fails to keep samples in distribution.
• Approximation of the intermediate posterior via Tweedie's formula introduces error that SMC corrects only asymptotically in particle number.

Experimental setup

Benchmarks: HPSv2 ImageReward

Metrics: target reward (aesthetic, PickScore) HPSv2 ImageReward Truncated CLIP Entropy LPIPS pairwise similarity Earth Mover's Distance on toy mixture of Gaussians

Baselines: DDPO AlignProp DPS FreeDoM MPGD DiffusionDPO SEIKO-UCB SEIKO-Bootstrap

Builds on: DPS TDS (Twisted Diffusion Sampler) DDPO AlignProp

Closest papers by meaning

• Inference-Time Alignment of Diffusion Models with Direct Noise Optimization (ICML 2025 · similarity 0.92)
• Diffusion Tree Sampling: Scalable inference‑time alignment of diffusion models (NeurIPS 2025 · similarity 0.90)
• Composition and Alignment of Diffusion Models using Constrained Learning (NeurIPS 2025 · similarity 0.90)
• Direct Distributional Optimization for Provable Alignment of Diffusion Models (ICLR 2025 · similarity 0.90)
• Hierarchical Variational Policies for Reward-Guided Diffusion (NeurIPS 2026 · similarity 0.90)
• Accelerating Convergence of Score-Based Diffusion Models, Provably (ICML 2024 · similarity 0.89)