Research map / Image generation / Fast diffusion sampling

Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Free Hunch, a training-free method that estimates denoiser covariance for diffusion models by transferring covariance across noise levels (Tweedie-based time update) and applying BFGS-style low-rank space updates from trajectory denoiser evaluations, improving linear inverse problem guidance especially at few steps.

Problem

Training-free guided generation with diffusion models needs accurate denoiser covariance Cov[x0|xt], but existing methods are either expensive (Jacobian/Hessian computation), require retraining or architecture changes, or make heavy approximations, leading to biased guidance at small step counts.

Approach

Combine a data-covariance initialization (DCT-diagonal) with a closed-form time update of the covariance via the Gaussian approximation of the diffusion marginals and BFGS-like low-rank space updates from consecutive denoiser mean evaluations, with a fallback to Sigma/sigma^2 Jacobian approximation when guidance is too large.

Key results

• At 15 Heun steps on Gaussian deblurring, FH+online reaches 23.54 PSNR / 0.378 LPIPS versus DPS 19.94 / 0.572 and ΠGDM 20.30 / 0.574.
• On random inpainting at 15 steps, FH+online achieves 25.25 PSNR and 0.317 LPIPS, versus best baseline TMPD 18.99 PSNR and 0.539 LPIPS for TMPD.
• FH gives the best LPIPS on most tasks at low step counts (15 and 30), with gains shrinking at higher step counts per the appendix.

Measured effects reported in the paper

Surprising findings

• Diagonal denoiser covariance (as in DPS and ΠGDM) causes guidance to be overestimated in high dimensions, making the generated variance too small and trajectories blow up without clipping.
• Using the Gaussian-approximation covariance in place of a heuristic identity gives large gains even though the fallback heuristics are often needed for other methods.

Limitations

• Focuses on linear inverse problems with Gaussian observation model; nonlinear problems left to future work.
• Time-update errors in low-dimensional settings can make covariance estimates rough; error bounds not derived.
• Relies on DCT-diagonal data covariance for images; applicability to other domains (e.g., PCA) unvalidated.

Experimental setup

Datasets: ImageNet Gaussian mixture toy data

Benchmarks: ImageNet 256x256 linear inverse problems (Gaussian deblurring, motion deblurring, random inpainting, 4x super-resolution

Metrics: PSNR SSIM LPIPS Jensen-Shannon divergence Frobenius norm of covariance error

Baselines: DPS ΠGDM TMPD Peng (Convert) Peng (Analytic) DDNM+ DiffPIR

Builds on: DPS ΠGDM Tweedie's formula BFGS quasi-Newton TMPD

Closest papers by meaning

• Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matching (ICLR 2025 · similarity 0.92)
• Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance (ICML 2024 · similarity 0.89)
• Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.89)
• O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.88)
• EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.88)
• Assessing the quality of denoising diffusion models in Wasserstein distance: noisy score and optimal bounds (NeurIPS 2025 · similarity 0.88)