Research map / Video / Video generation
Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies image diffusion models (SDXL) to temporally consistent video inverse problems (inpainting, 8x super-resolution) by combining Gaussian-process noise warping in function space with inference-time equivariance self-guidance.
Problem
Using image diffusion models frame-by-frame for video inverse problems causes flickering and temporal inconsistency, and prior noise-warping methods assume the denoiser is equivariant to spatial warps, which often fails for conditional tasks and latent diffusion models.
Approach
Treat frames as continuous functions and sample Gaussian-process noise (via random Fourier features) that can be evaluated at any warped location; fine-tune SDXL on GP noise so it works with correlated noise, and at inference guide the sampler with a self-equivariance loss that aligns the denoiser's output on the current warped frame with the previous frame's output.
Key results
- Warped Diffusion reduces mean latent/pixel warping error for translated inpainting to 0.001 versus 0.046 for How I Warped Your Noise (indep) and 0.061 for GP Warping, with FID 61.2.
- On 8x super-resolution of real FETV videos, Warped Diffusion achieves warping error 0.649 versus 0.812 for GP Warping and 0.934-1.048 for baselines, with FID 58.2 vs 47.6 for best-FID baseline.
- Fine-tuning SDXL on GP correlated noise matches independent-noise models on single-frame metrics (e.g., super-resolution FID 37.5 GP vs 40.8 independent).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Equivariance Self-Guidance to GP Noise Warping (ablation) | Warped Diffusion · COYO test, inpainting video translations | Warping Err: 0.061 {${\pm}$0.010} -> 0.001 {${\pm}$0.001} | Helped |
| Add Equivariance Self-Guidance to GP Noise Warping (ablation) | Warped Diffusion · COYO test, inpainting video translations | FID: 59.897 {${\pm}$3.718} -> 61.249 {${\pm}$2.499} | Mixed |
| Fine-tune SDXL with correlated GP noise instead of independent noise (ablation) | SDXL inpainting model · COYO test, single-frame inpainting | FID: 61.380 -> 58.727 | Helped |
| Fine-tune SDXL with correlated GP noise instead of independent noise (ablation) | SDXL super-resolution model · COYO test, single-frame super-resolution | FID: 40.843 -> 37.514 | Helped |
| Use GP Noise Warping with Equivariance Self-Guidance instead of GP Warping alone (ablation) | Warped Diffusion · FETV, 8× video super-resolution | Warping Err: 0.812${\pm}$ -> 0.649 {${\pm}$0.363} | Helped |
| Omit noise warping (ablation) | Warped Diffusion · Real-video super-resolution | restoration performance | Hurt |
Surprising findings
- Pre-finetuning SDXL produces unrealistic images when sampling is initialized with correlated noise, yet a short fine-tune restores quality.
- On real-video super-resolution, the best-FID baselines (Resample indep) achieve lower FID and LPIPS than Warped Diffusion despite higher temporal error.
Limitations
- Guidance increases sampling time substantially (about 5 minutes per 2-second video on one A100).
- Success depends on optical flow quality; occlusions and noisy flow cause flickering and failures in inpainting.
- Requires a model fine-tuned with correlated noise, so it cannot work zero-shot on arbitrary pretrained models.
Experimental setup
Datasets: COYO FETV
Benchmarks: COYO test split FETV
Metrics: FID Inception Score CLIP Text Score CLIP Image Score SSIM LPIPS MSE warping error
Baselines: Fixed noise Resample noise Nearest neighbor noise warping Bilinear interpolation noise warping How I Warped Your Noise GP Warping
Builds on: How I Warped Your Noise Stable Diffusion XL RAFT Diffusion Posterior Sampling Random Fourier Features
Closest papers by meaning
- Solving Video Inverse Problems Using Image Diffusion Models (ICLR 2025 · similarity 0.93)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.88)
- SITCOM: Step-wise Triple-Consistent Diffusion Sampling For Inverse Problems (ICML 2025 · similarity 0.87)
- NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer (ICLR 2025 · similarity 0.87)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.