Research map / Image generation / Text-to-image diffusion
Prompt-tuning Latent Diffusion Models for Inverse Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes P2L, which jointly tunes the null/text embedding on-the-fly during latent diffusion posterior sampling and projects latents onto the encoder's range via a proximal step, improving LDM-based inverse problem solving on FFHQ and ImageNet at 512x512.
Problem
Latent diffusion models for imaging inverse problems typically use a fixed null text prompt, which yields suboptimal reconstructions; LDM latents also drift off the encoder's range, causing artifacts.
Approach
Alternate between optimizing the text embedding on-the-fly to fit the measurement and updating the latent, plus periodically projecting the clean latent onto the range space of the VAE encoder via a proximal data-consistency step.
Key results
- On FFHQ SR x8, P2L reaches FID 31.23 vs 36.04 for the best LDIS baseline (LDIR) and LDPS 36.81.
- On FFHQ inpainting, P2L FID 26.27 vs 37.23 for LDIR-class baselines (about 10 FID gain in deblurring and inpainting).
- Ablation on FFHQ SR x8 shows prompt tuning and projection each improve FID (61.16 to 58.73 and 55.91 respectively), and combined reaches 52.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the text embedding on-the-fly instead of using the null prompt (ablation) | LDPS · FFHQ, SR ×8 | FID: 61.16 -> 58.73 | Helped |
| Add encoder-range projection while retaining prompt tuning (ablation) | P2L · FFHQ, SR ×8 | FID: 58.73 -> 52.96 | Helped |
| Add the proximal Γ update before encoder-range projection (ablation) | P2L with range projection · FFHQ, SR ×8 | FID: 55.91 -> 55.68 | Helped |
| Use prompt tuning, encoder-range projection, and proximal Γ together (ablation) | P2L · FFHQ inpainting | FID: 52.34 -> 46.92 | Helped |
| Replace proximal Γ with gluing (ablation) | P2L projection · FFHQ SR ×8, σ_y=0.05 | FID: 65.90 -> 68.99 | Hurt |
| Optimize prompt embeddings for 5 iterations with learning rate 1e-3 (ablation) | P2L prompt tuning · FFHQ SR ×8, 256 test images | FID: 61.16 -> 62.67 | Hurt |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · FFHQ motion deblurring | FID: 24.40 -> 28.34 | Hurt |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · FFHQ Gaussian deblurring | PSNR: 28.49 -> 26.97 | Hurt |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · ImageNet inpainting | PSNR: 23.28 -> 21.99 | Hurt |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · FFHQ SR ×8 | FID: 36.04 -> 31.23 | Helped |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · FFHQ inpainting | FID: 33.92 -> 26.27 | Helped |
| Use P2L instead of the strongest listed baseline | Stable Diffusion v1.4 inverse problem solver · ImageNet motion deblurring | FID: 59.08 -> 54.11 | Helped |
Surprising findings
- Optimized text embeddings sometimes outperform PALI captions generated from ground-truth images in FID.
- Successive encode-decode of the SD VAE does not reach a fixed point; images diverge even from clean natural images, undermining fixed-point regularization.
Limitations
- Prompt tuning adds extra forward/backward passes through LDM and decoder, increasing compute.
- Optimized continuous CLIP embeddings are not interpretable as discrete text.
- Classifier-free guidance not considered; PSNR slightly lower due to VAE imperfection.
Experimental setup
Datasets: FFHQ 512x512 ImageNet 512x512
Benchmarks: FFHQ 512x512 1k validation ImageNet 512x512 1k validation DIV2K 768x768
Metrics: FID LPIPS PSNR
Baselines: LDPS GML-DPS PSLD LDIR DDS DPS DiffPIR
Builds on: diffusion posterior sampling (DPS) latent diffusion models prompt tuning Adam PALI captioning
Closest papers by meaning
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.90)
- Regularization by Texts for Latent Diffusion Inverse Solvers (ICLR 2025 · similarity 0.90)
- Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency (ICLR 2024 · similarity 0.90)
- Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction (NeurIPS 2026 · similarity 0.90)
- Solving Inverse Problems with FLAIR (NeurIPS 2025 · similarity 0.90)
- Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.