Research map / Image generation / Image restoration
Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DISS, a training-free plug-and-play inference-time search (Greedy Search and Recursive Fork-Join Search) that injects side information via a reward-tilted posterior into diffusion inverse-problem solvers (DPS, DAPS, MPGD).
Problem
Diffusion-based inverse problem solvers ignore side information (reference images, text, anatomical MRI) that could greatly reduce ambiguity in severely ill-posed reconstructions; training conditional models for each side-information type is costly and inflexible.
Approach
Model the side-information posterior as a reward-tilted diffusion prior (p0 · exp(r/τ)) and sample it at inference time with particle-based search using grouped resampling (GS and RFJS), using black-box rewards such as face-identity similarity or text-image scores, with no retraining.
Key results
- On face box inpainting with DPS, RFJS reduces FaceSimilarity to 0.308 vs 0.739 for DPS (lower is better), with PSNR 28.29 vs 27.93.
- BON with side information improves DAPS box inpainting PSNR from 18.21 to 19.51, while BoN without side information (using measurement residual) drops it to 18.02.
- Search with side information consistently improves over DPS, RGG and BON across six linear and nonlinear deblurring/inpainting/super-resolution tasks.
Surprising findings
- Best-of-N selection by measurement residual without side information fails to improve and can degrade performance relative to the baseline.
- Classical metrics (PSNR, SSIM, LPIPS) can favor reconstructions with lower perceptual identity fidelity, so they fail to capture gains from side information.
Limitations
- Relies on quality of off-the-shelf reward models (e.g., AdaFace, ImageReward) and side information quality.
- Inference cost scales with the number of particles N, increasing compute relative to base solvers.
- Approximations (DPS-style value estimate, eta=0 for MPGD/DAPS) lack tight guarantees for the target posterior.
Experimental setup
Datasets: FFHQ CelebA-HQ ImageNet fastMRI knee
Benchmarks: FFHQ/CelebA-HQ face inverse problems ImageNet text side information fastMRI knee multi-contrast
Metrics: FaceSimilarity PSNR SSIM LPIPS CLIPScore NMI reward
Baselines: DPS BlindDPS DAPS MPGD Best-of-N RGG
Builds on: Diffusion Posterior Sampling (DPS) DAPS MPGD Best-of-N Reward gradient guidance
Closest papers by meaning
- Prompt-tuning Latent Diffusion Models for Inverse Problems (ICML 2024 · similarity 0.90)
- Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing (CVPR 2025 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.90)
- Learning Image Priors Through Patch-Based Diffusion Models for Solving Inverse Problems (NeurIPS 2024 · similarity 0.89)
- Solving Video Inverse Problems Using Image Diffusion Models (ICLR 2025 · similarity 0.89)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.