Research map / Image generation / Fast diffusion sampling
InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InvFusion, a training-based degradation-aware diffusion posterior sampler that injects the linear degradation operator into HDiT attention layers via Feature Degradation Layers, and also serves as an MMSE and NPPC predictor.
Problem
Diffusion-based inverse problem solvers face a trade-off: zero-shot methods accept any linear degradation but use approximations that reduce accuracy, while training-based posterior samplers are accurate but cannot condition on the degradation at test time (they solve blind restoration).
Approach
A Feature Degradation Layer applies the degradation operator H to internal network activations, concatenates the measurements y, applies a learned transform, and maps back via the pseudo-inverse; these are woven into the transformer via joint attention so one network is conditioned on H at test time.
Key results
- On FFHQ64 strided motion blur, InvFusion reaches FID 4.2 and CFID 4.2 with 63 NFEs, versus Palette's 4.7/5.8 and DPS's 9.7/17.7.
- On FFHQ64 matrix operator, InvFusion's MMSE predictor reaches 17.9 PSNR in a single NFE, versus Palette's 15.5.
- On ImageNet64 in-painting, InvFusion achieves FID 5.9 and CFID 5.9 versus Palette's 6.2 and 6.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition the trained sampler on the degradation operator using InvFusion blocks | Palette-style training-based sampler · FFHQ64 matrix operator | CFID: 25.5 -> 7.1 | Helped |
| Condition the trained sampler on the degradation operator using InvFusion blocks | Palette-style training-based sampler · FFHQ64 strided motion blur | PSNR: 21.0 -> 22.7 | Helped |
| Use InvFusion instead of Palette for MMSE prediction | FFHQ64 MMSE predictor · FFHQ64 strided motion blur | PSNR: 23.5 -> 25.0 | Helped |
| Use InvFusion instead of Palette for MMSE prediction | ImageNet64 MMSE predictor · ImageNet64 matrix operator | PSNR: 15.3 -> 17.3 | Helped |
| Use InvFusion instead of Palette under high Poisson noise | FFHQ64 restoration · Strided motion blur | FID: 17.6 -> 60.0 | Hurt |
| Apply InvFusion blocks at resolution 1 rather than none (ablation) | FFHQ64 sampler · Strided motion blur | CFID: 5.8 -> 4.5 | Helped |
| Apply InvFusion blocks at resolution 1 rather than none (ablation) | FFHQ64 sampler · Matrix operator | CFID: 25.5 -> 8.9 | Helped |
| Add InvFusion blocks at resolution 2 (ablation) | FFHQ64 sampler with resolution 1 blocks · Strided motion blur | CFID: 4.5 -> 4.4 | Helped |
| Add InvFusion blocks at resolution 2 (ablation) | FFHQ64 sampler with resolution 1 blocks · Matrix operator | CFID: 8.9 -> 7.9 | Helped |
| Add InvFusion blocks at resolution 3 (ablation) | FFHQ64 sampler with resolution 1 and 2 blocks · Matrix operator | CFID: 7.9 -> 7.1 | Helped |
| Add InvFusion blocks at resolution 3 (ablation) | FFHQ64 sampler with resolution 1 and 2 blocks · In-painting | CFID: 5.0 -> 5.0 | No clear effect |
| Train InvFusion as a diffusion model and use it for MMSE rather than training directly for MMSE | FFHQ64 InvFusion MMSE estimator · In-painting | PSNR: 19.79 -> 19.79 | No clear effect |
Surprising findings
- Conditional models trained with InvFusion or Palette outperform an unconditional baseline on unconditional generation, with InvFusion about 5% better.
- A diffusion model at high noise level (sigma_t=100) can serve as an MMSE estimator matching a model trained directly for MMSE.
Limitations
- Can only be trained on a finite set of degradation types and may under-perform beyond them.
- Only linear degradations are evaluated; non-linear generalization is untested.
- Computational and memory overhead from repeated degradation application during training.
Experimental setup
Datasets: FFHQ ImageNet
Benchmarks: FFHQ64 ImageNet64 FFHQ256
Metrics: PSNR FID CFID LPIPS NFEs MSE in degradation space
Baselines: DDRM DDNM DPS ΠGDM DAPS MGPS Palette InDI ASeq-DIP
Builds on: HDiT EDM2 Palette DPS DDNM
Closest papers by meaning
- Exact Posterior Score Estimation for Solving Linear Inverse Problems (NeurIPS 2026 · similarity 0.91)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.91)
- Analyzing and Guiding Zero-Shot Posterior Sampling in Diffusion Models (NeurIPS 2026 · similarity 0.91)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.91)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.90)
- Divide-and-Conquer Posterior Sampling for Denoising Diffusion priors (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.