Research map / Video / Video generation
PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pre-trained 512x512 latent text-to-video diffusion model for patch-wise video super-resolution via a dual-branch adapter (patch and global context with location mask), LoRA fine-tuning, and training-free multi-patch joint modulation, reaching 4K VSR efficiently.
Problem
Video diffusion models are pre-trained at a fixed token size (512x512), so full-frame VSR with them is computationally expensive (full attention) and locked to fixed output resolution; patch-level detail generation is also out of their training distribution.
Approach
Split the upscaled video into patches matching the base model's generation size, enhance each with a dual-stream adapter (patch features plus a resized full-video global context fused via cross-attention, with a binary patch-location mask), fine-tune the base with LoRA, and stitch patches using auxiliary overlapping patches with weighted fusion (multi-diffusion style) to avoid seams.
Key results
- On REDS30, PatchVSR achieves the best LPIPS (0.326) and MUSIQ (44.956), though with lower PSNR (30.111) than Upscale-A-Video (31.257).
- On VideoGen30, PatchVSR reaches DOVER 0.590 and MUSIQ 50.559, the top MUSIQ among compared methods.
- For 2K video generation, PatchVSR takes 680 s and 40 GB versus 2261 s/68 GB for LaVie-SR and 1562 s/62 GB for VEnhancer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a global context branch (ablation) | PatchVSR · VideoGen30 | DOVER: 0.502 -> 0.590 | Helped |
| add a global context branch (ablation) | PatchVSR · VideoGen30 | MUSIQ: 46.074 -> 50.559 | Helped |
| add LoRA fine-tuning (ablation) | PatchVSR · VideoGen30 | DOVER: 0.582 -> 0.590 | Helped |
| add patch location embedding (ablation) | PatchVSR global branch · VideoGen30 | DOVER: 0.574 -> 0.590 | Helped |
| use a fixed prompt in the patch condition branch (ablation) | PatchVSR · VideoGen30 | MUSIQ: 48.133 -> 50.559 | Helped |
| use a fixed prompt in the patch condition branch (ablation) | PatchVSR · VideoGen30 | DOVER: 0.562 -> 0.590 | Helped |
| use multi-patch joint modulation with auxiliary bridging patches and spatial weights (ablation) | PatchVSR · 2K video super-resolution | seam artifacts | Helped |
| use PatchVSR instead of the strongest baseline | video super-resolution · REDS30 | MUSIQ: 44.533 -> 44.956 | Helped |
| use PatchVSR instead of the strongest baseline | video super-resolution · REDS30 | PSNR: 31.257 -> 30.111 | Hurt |
| use PatchVSR instead of the strongest baseline | video super-resolution · SynVideo30 | LPIPS: 0.185 -> 0.183 | Helped |
| use PatchVSR instead of the strongest baseline | video super-resolution · SynVideo30 | PSNR: 33.507 -> 30.857 | Hurt |
| use PatchVSR instead of the strongest baseline | video super-resolution · VideoGen30 | MUSIQ: 48.329 -> 50.559 | Helped |
Surprising findings
- Higher PSNR and SSIM are not achieved by the method; the authors argue the richer generated details penalize pixel-wise metrics.
- Replacing text prompts with a fixed prompt in the patch branch improves results (DOVER 0.562 with text vs 0.590 with fixed prompt in ablation).
Limitations
- Noticeable performance degradation on real-world low-resolution inputs due to lack of specialized degradation augmentation.
- Inherits iterative multi-step diffusion inference, which is slow for practical use.
- Inferior PSNR and SSIM compared with several baselines.
Experimental setup
Datasets: 460K self-collected video-text pairs REDS30 SynVideo30 VideoGen30
Benchmarks: REDS30 SynVideo30 VideoGen30
Metrics: PSNR SSIM LPIPS DOVER MUSIQ Aesthetics Time Memory
Baselines: VEnhancer Upscale-A-Video LaVie-SR RealBasicVSR
Builds on: Rectified Flow multi-diffusion LoRA VEnhancer
Closest papers by meaning
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.92)
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.89)
- SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution (NeurIPS 2024 · similarity 0.88)
- SkipSR: Faster Super-Resolution with Token Skipping (NeurIPS 2026 · similarity 0.88)
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (CVPR 2024 · similarity 0.88)
- VideoGigaGAN: Towards Detail-rich Video Super-Resolution (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.