Research map / Image generation / Fast diffusion sampling
Gradient-free Decoder Inversion in Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gradient-free decoder inversion for latent diffusion models using a forward step method on the encoder-decoder composition, with convergence theory (including inertial KM iterations) and Adam plus cosine learning-rate scheduling for practical speed and memory savings.
Problem
Decoder inversion in latent diffusion models (finding a latent that decodes to a given image) is typically solved with gradient descent through the decoder, which is memory- and time-intensive for large or video latents.
Approach
Replace backpropagation through the decoder with a gradient-free forward step / fixed-point iteration z <- z - rho(E(D(z)) - E(x)) that uses the off-the-shelf encoder, with convergence proofs under a cocoercivity assumption and an inertial KM variant; practical version uses Adam and cosine learning-rate scheduling in 16-bit.
Key results
- Gradient-free inversion reaches about -16.4 dB in 1.89 s vs 9.51 s for gradient-based on InstaFlow, up to 5x faster.
- On LaVie with 16-bit, gradient-free reaches -21.37 dB in 21.19 s using 7.13 GB, vs gradient-based -19.06 dB in 25.1 s using 64.7 GB.
- Watermark classification on SD2.1 gives 202/300 accuracy vs 207/300 for gradient-based, with peak memory 6.35 GB vs 11.4 GB and runtime 22.9 s vs 38.0 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace gradient-based inversion with gradient-free inversion | Decoder inversion · InstaFlow, 20 iterations | Runtime (s): 6.30 s -> 1.89 s | Helped |
| Replace gradient-based inversion with gradient-free inversion | Decoder inversion · LaVie | Peak memory: 64.7 GB -> 7.13 GB | Helped |
| Replace gradient-based inversion with gradient-free inversion | Decoder inversion · InstaFlow, 20 iterations | NMSE (dB): -15.94 -> -16.38 | Helped |
| Replace gradient-based inversion with gradient-free inversion | Decoder inversion · SD2.1 tree-rings watermark classification | Accuracy: 207/300 -> 202/300 | Hurt |
| Replace gradient-based inversion with gradient-free inversion | Decoder inversion · InstaFlow tree-rings watermark classification | Accuracy: 227/300 -> 227/300 | No clear effect |
| Use Adam instead of vanilla forward step (ablation) | Gradient-free decoder inversion · Stable Diffusion 2.1, 20 iterations | NMSE (dB): -16.87 ± 0.38 -> -19.39 ± 0.54 | Helped |
| Use Adam instead of inertial KM iterations (ablation) | Gradient-free decoder inversion · Stable Diffusion 2.1, 20 iterations | NMSE (dB): -18.99 ± 0.53 -> -19.39 ± 0.54 | Helped |
| Use scheduled LR instead of fixed LR 0.01 (ablation) | Adam decoder inversion · Stable Diffusion 2.1, 100 iterations | NMSE (dB): -21.22 ± 0.74 -> -21.71 ± 0.77 | Helped |
| Use scheduled LR instead of fixed LR 0.002 (ablation) | Adam decoder inversion · Stable Diffusion 2.1, 100 iterations | NMSE (dB): -20.59 ± 0.64 -> -21.71 ± 0.77 | Helped |
| Use scheduled LR instead of fixed LR 0.01 (ablation) | Adam decoder inversion · Stable Diffusion 2.1, 200 iterations | NMSE (dB): -20.61 ± 0.79 -> -21.85 ± 0.82 | Helped |
| Use scheduled LR instead of fixed LR 0.002 (ablation) | Adam decoder inversion · Stable Diffusion 2.1, 200 iterations | NMSE (dB): -21.57 ± 0.74 -> -21.85 ± 0.82 | Helped |
Surprising findings
- Gradient-free inversion with Adam in 16-bit achieves comparable accuracy, whereas gradient-based 16-bit is infeasible due to underflow.
- Cocoercivity assumption empirically holds for most instances, and better convergence correlates with more cocoercive behavior.
Limitations
- Gradient-based inversion is more accurate when sufficient runtime and GPU memory are available.
- Convergence guarantee does not cover the Adam and learning-rate-scheduled variant used in practice.
- Decoder inversion is often unnecessary for applications like background-preserving editing where encoder alone suffices.
Experimental setup
Datasets: Gustavosta/Stable-Diffusion-Prompts
Benchmarks: tree-rings watermark classification (300 images) decoder inversion NMSE on generated images (100 instances per LDM)
Metrics: NMSE (dB) runtime (s) peak memory (GB) watermark classification accuracy
Baselines: Gradient-based decoder inversion (inv-DM) vanilla gradient descent decoder inversion encoder-only naive DDIM inversion backward Euler inversion
Builds on: Tree-rings watermarking Adam inertial Krasnoselskii-Mann iteration gradient-based GAN inversion DDIM inversion
Closest papers by meaning
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.90)
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.89)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.89)
- OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.89)
- Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding (ICML 2024 · similarity 0.88)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.