Research map / Image generation / Autoregressive and latent generation
High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiFI, a patch-based cascaded pixel diffusion model for high-resolution (up to 8K) frame interpolation that reuses one fixed-resolution denoiser across cascade levels via patch-wise MultiDiffusion merging.
Problem
Video frame interpolation at extremely high resolution (2K to 8K) struggles with large motion, repetitive textures, and thin objects, and standard cascaded diffusion becomes memory-prohibitive at such resolutions.
Approach
A patch-based cascade that always denoises at a fixed resolution, upsampling by processing overlapping patches of the inputs and the prior coarse prediction with MultiDiffusion merging; a single model is shared across base and super-resolution stages via image-level conditioning dropout.
Key results
- On X-TEST 4K, HiFI reaches 32.92 dB PSNR (0.931 SSIM) with the raw-video variant, versus 31.46 dB for EMA-VFI and 30.81 dB for M2M.
- On SEPE-8K, HiFI achieves 29.78 dB PSNR, while M2M gets 28.34 dB and SGM-VFI 28.43 dB; most other baselines run out of memory.
- Patch-based cascade keeps peak memory at 7.53 GB for 4K inputs, versus OOM for whole-image processing, at the cost of 164 s inference time per sample.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add raw videos to Vimeo + X-TRAIN training data | HiFI · X-TEST 4K | PSNR: 32.03 -> 32.92 | Helped |
| add the shared-model self-cascade instead of using only the base model (ablation) | HiFI · X-TEST 4K | PSNR: 28.57 -> 32.92 | Helped |
| use a shared model for a three-stage self-cascade instead of separate models in a standard two-stage cascade (ablation) | HiFI · X-TEST 4K | PSNR: 32.48 -> 32.92 | Helped |
| increase the self-cascade from two to three stages (ablation) | HiFI · X-TEST 4K | PSNR: 32.77 -> 32.92 | Helped |
| replace coarse-to-fine tiling refinement with HiFI self-cascade (ablation) | HiFI · X-TEST 4K | PSNR: 32.54 -> 32.92 | Helped |
| add attention layers (ablation) | HiFI base model · X-TEST 4K | PSNR: 27.75 -> 28.57 | Helped |
| add the diffusion process (ablation) | HiFI base model · X-TEST 4K | PSNR: 27.56 -> 28.57 | Helped |
| use 32 rather than 4 sampling steps (ablation) | HiFI self-cascade · X-TEST 4K | PSNR: 32.92 -> 32.68 | No clear effect |
| increase patch size from 256×384 to 512×768 (ablation) | HiFI self-cascade · X-TEST 4K | PSNR: 32.70 -> 32.92 | No clear effect |
| increase overlap ratio from 1.33 to 2.13 (ablation) | HiFI self-cascade · X-TEST 4K | PSNR: 32.92 -> 33.10 | No clear effect |
| add image-level dropout during cascade training (ablation) | HiFI self-cascade · challenging large-motion scenes | fine-detail preservation | Helped |
| increase sampling steps from 4 to 8 (ablation) | HiFI base model · X-TEST 4K | PSNR: 28.57 -> 29.67 | Helped |
Surprising findings
- A dedicated super-resolution model trained without conditioning dropout ignores input frames and shortcuts by sharpening the low-resolution estimate.
- Using one shared model for base and cascade stages performs on par with or better than a dedicated upsampling model with twice the parameters.
Limitations
- Slow inference (164 s per 4K sample) and heavy model size (647M parameters) limit practical use.
- Struggles with fluid motion, producing blurry outputs, and with motion larger than the inference patch size (e.g., ~1500 px).
- Accuracy on Vimeo-90K is only comparable to state of the art because the model predicts non-linear motion that differs from the dataset's linear-motion bias.
Experimental setup
Datasets: Vimeo-90K X-TRAIN large-scale internet video dataset (30M videos) SEPE-8K Xiph X-TEST
Benchmarks: Vimeo-90K Xiph 2K Xiph 4K X-TEST 2K X-TEST 4K SEPE-8K LaMoR
Metrics: PSNR SSIM LPIPS inference time peak memory
Baselines: M2M FILM AMT UPR-Net FITUG TCL IQ-VFI EMA-VFI XVFI BiFormer SGM-VFI LDMVFI
Builds on: DDVM CDM MultiDiffusion v-parameterization
Closest papers by meaning
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.88)
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.88)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.88)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.88)
- Video Interpolation with Diffusion Models (CVPR 2024 · similarity 0.87)
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.