Research map / Image generation / Fast diffusion sampling
DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised $h$-transform
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies conditional diffusion sampling under Doob's generalised h-transform and proposes DEFT, which freezes a large unconditional diffusion model and fine-tunes a small network to learn the conditional h-transform via denoising score matching.
Problem
Conditional sampling from pretrained unconditional diffusion models for inverse problems is heuristic, sensitive to hyperparameters, and expensive; methods often need backpropagation through large score networks or access to weights behind closed APIs.
Approach
Cast conditional diffusion sampling as a generalised Doob's h-transform, and learn the guidance term h directly from paired data with a small network that takes the unconditional Tweedie estimate and a likelihood gradient as inputs, leaving the unconditional score network frozen and never backpropagating through it.
Key results
- On ImageNet inpainting, DEFT achieves the best LPIPS (0.09) and KID (0.29) among compared methods, with 5.2 hrs total vs 30.72 hrs for DPS, including training time.
- On non-linear deblurring, DEFT reaches PSNR 25.16 and SSIM 0.79 versus RED-diff's 21.21 and 0.53.
- On motif scaffolding, DEFT solves 10 of 12 tasks versus 5 for DPS, using a 200k-parameter h-transform (vs 4.1M base Genie).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DEFT instead of RED-diff | ImageNet inpainting · ImageNet inpainting | KID: 0.86 -> 0.29 | Helped |
| Use DEFT instead of RED-diff | ImageNet inpainting · ImageNet inpainting | PSNR: 23.29 -> 22.18 | Hurt |
| Use DEFT instead of RED-diff | ImageNet 4x super-resolution · ImageNet 4x super-resolution | KID: 10.0 -> 1.78 | Helped |
| Use DEFT instead of RED-diff | ImageNet 4x super-resolution · ImageNet 4x super-resolution | PSNR: 25.95 -> 24.92 | Hurt |
| Use DEFT instead of RED-diff | ImageNet HDR · ImageNet HDR | PSNR: 25.23 -> 28.51 | Helped |
| Use DEFT instead of RED-diff | ImageNet phase retrieval · ImageNet phase retrieval | PSNR: 10.53 -> 13.03 | Helped |
| Use DEFT instead of RED-diff | ImageNet non-linear deblurring · ImageNet non-linear deblurring | PSNR: 21.21 -> 25.16 | Helped |
| Use DEFT instead of RED-diff | AAPM CT reconstruction · AAPM CT | PSNR: 34.85 -> 34.73 | Hurt |
| Use DEFT instead of RED-diff | LoDoPab-CT reconstruction · LoDoPab-CT | PSNR: 34.95 -> 35.81 | Helped |
| Use DEFT instead of DPS | Genie motif scaffolding · Motif scaffolding (12 contiguous targets) | tasks solved: 5 tasks -> 10 out of the 12 tasks | Helped |
| Increase h-transform training set from 10 to 1000 images (ablation) | DEFT · ImageNet inpainting | KID: 1.85 -> 0.29 | Helped |
| Increase h-transform training set from 10 to 1000 images (ablation) | DEFT · ImageNet inpainting | PSNR: 20.87 -> 22.18 | Helped |
Surprising findings
- DPS performs poorly on the HDR task (PSNR 7.94), and in phase retrieval all methods show highly variable quality across seeds.
- The original RED-diff non-linear deblurring forward operator leads to a nearly trivial reconstruction task, according to the authors.
Limitations
- Requires a small paired fine-tuning dataset, which risks overfitting to data biases, unlike zero-shot methods such as DPS and ΠGDM.
- The zero-shot stochastic control variant is computationally expensive for high-dimensional data and was only shown on MNIST.
- Phase retrieval results remain inconsistent across seeds, with room for improvement.
Experimental setup
Datasets: ImageNet AAPM 2016 grand challenge CT LoDoPab-CT SCOPe Flowers
Benchmarks: ImageNet 256x256 inpainting ImageNet 4x super-resolution HDR phase retrieval non-linear deblurring AAPM CT LoDoPab-CT RFDiffusion motif scaffolding benchmark
Metrics: PSNR SSIM KID LPIPS top-1 accuracy sampling time scRMSD motifRMSD
Baselines: DPS ΠGDM DDRM RED-diff I2SB Genie amortised training
Builds on: DPS RED-diff Genie I2SB DDPM
Closest papers by meaning
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.92)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.90)
- EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.90)
- Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising (NeurIPS 2024 · similarity 0.90)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.90)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.