Research map / Image generation / Image restoration

Learning Diffusion Texture Priors for Image Restoration

CVPR 2024 · Image generation › Image restoration · approach: Diffusion-based universal restoration · 33 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DTPM, a diffusion model pre-trained on ~55K smoothed/texture-residual pairs to learn texture priors, then adapted to image restoration tasks via conditional guidance adapters and an initial U-Net predictor.

Problem

Diffusion models for image restoration struggle to preserve content fidelity due to sampling randomness, while deterministic regression models produce weak fine-grained textures.

Approach

Pre-train a diffusion model to predict the residual texture layer (high-quality image minus edge-preserving smoothed version) conditioned on a ResNet-18 semantic code, then freeze most weights and insert conditional guidance adapters fed by an initial U-Net predictor output for downstream restoration.

Key results

• On GoPro motion deblurring, DTPM-50 Step reaches LPIPS 0.073 and FID 7.69 (text) while DTPM-4 Step gets FID 10.02 vs DvSR 12.20; PSNR drops to 31.11 at 50 steps.
• On Outdoor-Rain deraining/dehazing, DTPM-4 Step achieves PSNR 30.99, SSIM 0.934, LPIPS 0.0635, outperforming RainHazeDiff's 28.38 PSNR.
• Ablation on Outdoor-Rain shows removing the initial predictor drops PSNR from 30.99 to 26.91 and removing Stage I pretraining drops it to 28.90.

Measured effects reported in the paper

Surprising findings

• Removing the semantic code slightly improves PSNR (29.79 vs 30.99 is lower; reported as improving quality in text) and LPIPS trade-offs, which the authors attribute to possible informative features.
• More sampling steps trade PSNR for LPIPS; PSNR falls from just above 31 dB while LPIPS improves, and on desnowing the 4-step variant gives best PSNR.

Limitations

• Sampling still requires multiple steps and is not fast; authors note need to accelerate sampling and reduce model size.
• Pixel-level fidelity (PSNR) degrades as sampling steps increase, creating a distortion-perception trade-off.
• Stage I pretraining and initial predictor add architectural complexity; evaluation reported on a limited set of restoration tasks.

Experimental setup

Datasets: DIV2K Flickr2K GoPro DPDD Snow100K Outdoor-Rain Raindrop

Benchmarks: Snow100K-S Snow100K-L Outdoor-Rain RainDrop DPDD GoPro

Metrics: PSNR SSIM LPIPS FID MAE

Baselines: Restormer MPRNet NAFNet SnowDiff RainHazeDiff RainDropDiff DvSR IFAN KPAC IDT

Builds on: DDIM NAFNet WeatherDiff DvSR Restormer

Closest papers by meaning

• A Modular Conditional Diffusion Framework for Image Reconstruction (NeurIPS 2024 · similarity 0.89)
• FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
• RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution (AAAI 2025 · similarity 0.88)
• Diffusion Priors for Variational Likelihood Estimation and Image Denoising (NeurIPS 2024 · similarity 0.88)
• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration (ICLR 2025 · similarity 0.88)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.88)