Research map / Video / Video generation
Solving Video Inverse Problems Using Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Solves video inverse problems (temporal and spatio-temporal degradations) using only pretrained image diffusion models by treating time as batch dimension, with batch-consistent noise sampling and multi-step CG data-consistency optimization on denoised batches.
Problem
Video inverse problems with spatio-temporal degradation (temporal PSF blur plus spatial deblurring, super-resolution, inpainting) are underexplored because training video diffusion models is costly.
Approach
Treat the video's time axis as the batch axis of a pretrained image diffusion model, perform Krylov-subspace (CG) data-consistency optimization jointly over the denoised spatio-temporal batch at each reverse step, and synchronize the initial and re-noising stochastic noise across frames to keep temporal consistency.
Key results
- On DAVIS with uniform PSF k=7, the method reaches 43.16 PSNR and 0.008 FVD in 12 s with 20 NFE, versus 33.42 PSNR for DPS (1000 NFE, 1244 s).
- On DAVIS 4x super-resolution with temporal degradation, the method reaches 25.71 PSNR and 0.352 FVD (x10^-3) vs. 21.39 PSNR for DPS.
- Human study (36 votes) preferred the method for quality and temporal consistency at 94.4% of votes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use batch-consistent image diffusion sampling with spatio-temporal CG updates instead of DPS | Video inverse reconstruction · DAVIS, temporal uniform PSF (k=7) | PSNR: 33.42 -> 43.16 | Helped |
| Use batch-consistent image diffusion sampling with spatio-temporal CG updates instead of DiffusionMBIR | Video inverse reconstruction · DAVIS, temporal uniform PSF (k=13) | FVD: 0.836 -> 0.035 | Helped |
| Use the proposed solver instead of ADMM-TV | Video inverse reconstruction · DAVIS, temporal uniform PSF (k=7) + deblurring | PSNR: 22.76 -> 27.77 | Helped |
| Use the proposed solver instead of DiffusionMBIR | Video inverse reconstruction · DAVIS, temporal uniform PSF (k=7) + 4× super-resolution | PSNR: 21.41 -> 25.71 | Helped |
| Use the proposed solver instead of DPS | Video inverse reconstruction · DAVIS, temporal uniform PSF (k=7) + inpainting | PSNR: 27.43 -> 29.45 | Helped |
| Add the proposed blind reconstruction stages to DeepDeblur | Video deblurring · GoPro test, blind deblurring | PSNR: 30.93 -> 38.98 | Helped |
| Use the proposed solver instead of DiffIR2VR | 4× video super-resolution · DAVIS | PSNR: 30.51 -> 32.88 | Helped |
| Use the proposed solver instead of DiffIR2VR | 4× video super-resolution · Pexels | PSNR: 31.31 -> 33.79 | Helped |
| Add stochastic noise rather than eliminating it (η=0.0; compare η=0.8) (ablation) | Proposed video inverse solver · DAVIS, temporal degradation + 4× super-resolution | PSNR: 18.04 -> 25.71 | Helped |
| Vary η from 0.0 to 1.0 (ablation) | Proposed video inverse solver · DAVIS, temporal degradation + 4× super-resolution | PSNR: 18.04 -> 26.04 | Helped |
| Add diffusion updates to standalone CG reconstruction (ablation) | Video inverse solver · Spatio-temporal degradations | reconstruction quality | Helped |
| Synchronize stochastic noise across the batch instead of using no stochasticity control (ablation) | Proposed video inverse solver · Video inverse reconstruction | inter-batch difference | Helped |
Surprising findings
- Using only an unconditional image diffusion model without any video prior, the method outperforms video-consistent baselines such as DiffusionMBIR and DPS on temporal and spatio-temporal degradations.
Limitations
- Limited to 256x256 resolution because it uses a pixel-space unconditional diffusion model.
- Less reliable under severe temporal degradation such as frame interpolation.
- Blind deblurring requires two sampling rounds, doubling sampling time.
Experimental setup
Datasets: DAVIS 2017 GoPro
Benchmarks: DAVIS 2017 GoPro
Metrics: PSNR SSIM LPIPS FVD runtime
Baselines: DPS DiffusionMBIR ADMM-TV stand-alone CG DiffIR2VR
Builds on: decomposed diffusion sampler diffusion posterior sampling DiffusionMBIR DDIM
Closest papers by meaning
- Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models (NeurIPS 2024 · similarity 0.93)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.91)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.91)
- SITCOM: Step-wise Triple-Consistent Diffusion Sampling For Inverse Problems (ICML 2025 · similarity 0.90)
- Improving Diffusion-Based Image Restoration with Error Contraction and Error Correction (AAAI 2024 · similarity 0.90)
- Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.