Research map / Image generation / Fast diffusion sampling
Regularization by Texts for Latent Diffusion Inverse Solvers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TReg, a training-free latent diffusion inverse solver that uses text descriptions of the desired solution as regularization during reverse sampling, with adaptive negation (null-text optimization via CLIP similarity) to reduce ambiguity in inverse problems.
Problem
Inverse problems such as Fourier phase retrieval and severely degraded super-resolution, deblurring and inpainting are ill-posed; diffusion inverse solvers rely only on image statistics and fail to break symmetries or resolve ambiguity, and existing latent solvers underuse text conditioning.
Approach
Treat the text-conditioned Tweedie estimate as a pivot in a proximal latent MAP objective, solved by variable splitting with a closed-form interpolation in latent space, and update the CFG null-text embedding adaptively by minimizing its CLIP similarity to the current image estimate (adaptive negation).
Key results
- On ice-cream Food-101 super-resolution x16, TReg (w/ adaptive negation) reaches 124.2 FID vs 176.1 for PSLD+CFG and 280.1 for PSLD, with PSNR 21.09 vs 17.73 for PSLD.
- On FFHQ with ground-truth class text, TReg attains the best FID on deblurring (40.24 vs 59.41 for PSLD) and the best PSNR on SR x16 among the compared methods (22.60 vs 21.94 for P2L).
- Runtime for linear problems is 0.26 min for TReg vs 1.47 min for PSLD and 2.00 min for P2L at NFE=200.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive negation (ablation) | TReg · Food-101, Ice Cream and Fried Rice, SRx16 and deblurring | PSNR: 22.05 -> 21.09; 21.26 -> 19.98; 23.91 -> 23.73; 22.99 -> 22.83 | Hurt |
| add adaptive negation (ablation) | TReg · Food-101, Ice Cream and Fried Rice, SRx16 and deblurring | FID: 144.4 -> 124.2; 131.2 -> 120.6; 132.0 -> 97.40; 133.4 -> 112.1 | Helped |
| use a text description instead of null text | TReg · blurry-image reconstruction, ImageNet validation set | pixel-level variance | Helped |
| use null text instead of a solution-describing prompt | TReg · inverse problem reconstruction | ambiguity reduction | Hurt |
| vary prompt wording among "a cat," "a photo of a cat," and "a photography of a cat" (ablation) | TReg · AFHQ-cat super-resolution | reconstruction sensitivity to prompt wording | No clear effect |
| add text regularization to Latent DPS | Fourier phase retrieval · Fourier phase retrieval | symmetry breaking | Helped |
| use text conditioning through CFG in Latent DPS | Latent DPS · non-uniform deblurring with gamma correction | reconstruction quality | Hurt |
| add text regularization | Latent DPS · non-uniform deblurring with gamma correction | reconstruction quality | Helped |
| add text regularization | inverse-problem reconstruction · JPEG-compressed measurements, quality factors Q50 and Q30 | reconstruction robustness | No clear effect |
| add text regularization | inverse-problem reconstruction · JPEG-compressed measurements, quality factor Q10 | reconstruction quality | Hurt |
Surprising findings
- Without adaptive negation, TReg achieves higher PSNR but worse FID, as the reconstructions become blurrier with missing details.
- PSLD with CFG achieves lower LPIPS in some text-mismatch cases but loses data consistency, showing large y-MSE values.
Limitations
- Fourier phase retrieval requires an Adam optimizer in pixel space, adding computational cost compared with the CG-based linear case.
- Residual uncertainties remain in details such as head position or mouth shape even with text regularization.
- Text prompt choice must be carefully set; a mismatched class (e.g., 'dog' for a baby image) cannot effectively reduce the solution space.
Experimental setup
Datasets: Food-101 FFHQ AFHQ ImageNet LHQ
Benchmarks: Food-101 FFHQ AFHQ ImageNet
Metrics: PSNR FID LPIPS CLIP similarity y-MSE
Baselines: DDRM PGDM PSLD PSLD + CFG P2L PnP PSLD+DDS PSLD+PnP Repaint Stable-Inpaint
Builds on: Latent Diffusion Models PSLD P2L Plug-and-Play diffusion Delta Denoising Score DDIM
Closest papers by meaning
- Prompt-tuning Latent Diffusion Models for Inverse Problems (ICML 2024 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.89)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.89)
- SITCOM: Step-wise Triple-Consistent Diffusion Sampling For Inverse Problems (ICML 2025 · similarity 0.87)
- Self-diffusion for Solving Inverse Problems (NeurIPS 2025 · similarity 0.87)
- DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.