Research map / Optimization / Optimizers
Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMILO and DMILO-PGD, which combine intermediate layer optimization (per-sampling-step optimization with sparse deviations) and projected gradient descent with DMPlug-style diffusion priors to cut memory and avoid suboptimal convergence in diffusion-based inverse problem solvers.
Problem
Diffusion-model-based solvers for inverse problems (CSGM-type, e.g., DMPlug) suffer from heavy memory use because gradients must traverse the full multi-step sampling chain, and from suboptimal convergence depending on initialization.
Approach
Treat each DDIM sampling step as a separate function and optimize latent inputs layer-by-layer (intermediate layer optimization) with a sparse deviation term, so only one step's gradient graph is stored; DMILO-PGD wraps this in projected gradient descent that projects using the forward operator.
Key results
- On 4x super-resolution on CelebA, DMILO-PGD reaches 33.58 PSNR and 0.906 SSIM, versus the second-best method at about 32.48 PSNR and 0.888 SSIM.
- On inpainting (70% random mask) on FFHQ, DMILO-PGD achieves 34.62 PSNR? no: DMILO-PGD reports 36.42 PSNR and 0.952 SSIM on CelebA inpainting, improving on the best competitor by about 0.91 dB.
- Memory for DMPlug grows from 10.53 GB (1 step) to N/A at 4 steps, while DMILO and DMILO-PGD stay at about 10.54 GB across steps (2.75 GB model, RTX 4090).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMILO-PGD instead of competing methods | Nonlinear deblurring · Nonlinear deblurring, CelebA and FFHQ | PSNR: +2.55 PSNR; +2.65 PSNR | Helped |
| Use DMILO rather than DMILO-PGD | Blind image deblurring · BID (Gaussian and motion kernels) | reconstruction quality | Mixed |
| Add sparse deviations (ablation) | DMILO · Super-resolution, CelebA | LPIPS, PSNR, and SSIM | Helped |
| Optimize through only the last timestep rather than all diffusion steps (ablation) | DMILO · Inpainting, CelebA | reconstruction performance | Hurt |
| Use intermediate-layer optimization to optimize one sampling step at a time | DMILO vs. DMPlug · Super-resolution, LSUN Bedroom | memory usage (GB): 20.83 -> 10.53; N/A -> 10.54 | Helped |
Surprising findings
- DMILO-PGD underperforms on blind image deblurring, which the authors attribute to the naive gradient update being unsuited to updating the blur kernel.
- Optimizing only the last sampling step (LTS) remains somewhat effective but degrades reconstruction quality relative to full multi-step optimization.
Limitations
- DMILO-PGD is less effective for blind deblurring, where kernel updates are not well handled by naive gradient steps.
- Theoretical analysis is restricted to a two-function generator composition and linear measurements with Gaussian matrices.
- Initialization strategies for last-timestep-only optimization remain an open challenge.
Experimental setup
Datasets: CelebA FFHQ LSUN-Bedroom ImageNet
Benchmarks: CelebA FFHQ ImageNet LSUN-Bedroom
Metrics: PSNR SSIM LPIPS FID
Baselines: DDRM DPS ΠGDM RED-diff DAPS DiffPIR DMPlug MGPS DCPS DPIR ILVR Blind-DPS
Builds on: DMPlug Intermediate Layer Optimization CSGM Projected gradient descent for generative priors DDIM
Closest papers by meaning
- DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models (NeurIPS 2024 · similarity 0.93)
- Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems (NeurIPS 2024 · similarity 0.90)
- Improving Diffusion-based Inverse Algorithms under Few-Step Constraint via Linear Extrapolation (NeurIPS 2025 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.90)
- InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems (NeurIPS 2025 · similarity 0.90)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.