Research map / Image generation / Autoregressive and latent generation
Hierarchical Patch Diffusion Models for High-Resolution Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hierarchical patch diffusion models (HPDM) that train end-to-end on high-resolution video by denoising coarse-to-fine patch hierarchies with deep context fusion and adaptive computation, achieving SOTA UCF-101 FVD and fast fine-tuning for text-to-video.
Problem
Diffusion models are hard to scale end-to-end to high-resolution images and videos; existing pipelines (latent or cascaded diffusion) use multiple components, and prior patch-wise diffusion still needs full-resolution training steps.
Approach
Model only a hierarchy of small random patches at exponentially decreasing scales, conditioning each finer patch on spatially aligned, grid-sampled features from all coarser pyramid levels (deep context fusion), and skip early blocks for high-resolution levels (adaptive computation), with tiled MultiDiffusion-style inference.
Key results
- HPDM-L reaches FVD 66.32 and IS 87.68 on UCF-101 16x256x256 class-conditional generation, more than 100% better than prior methods without pretraining.
- Deep context fusion vs shallow fusion: FVD@512 at 64x256x256 drops from 467.0 to 378.2 in the ablation.
- HPDM-T2V is fine-tuned from a 36x64 SnapVideo base for only 15k steps and reaches zero-shot UCF-101 FVD 299.3 at 72x128 resolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HPDM-L instead of recent UCF-101 generators | UCF-101 class-conditional video generation · UCF-101, 16×256² | FVD: 81.25 -> 66.32 | Helped |
| add deep context fusion instead of shallow RGB-pixel fusion (ablation) | HPDM · UCF-101, 64×256² | FVD@512: 467.0 -> 378.2 | Helped |
| allow gradients to flow through aggregated context features (ablation) | HPDM · UCF-101, 64×256² | FVD@512: 397.3 -> 378.2 | Helped |
| add adaptive computation by skipping early blocks at higher pyramid levels (ablation) | HPDM · UCF-101, 64×256² | training speed (videos/sec): 2.73 -> 4.4 | Helped |
| add adaptive computation (ablation) | HPDM · UCF-101, 64×256² | FVD@512: 373.9 -> 378.2 | Mixed |
| include patch coordinates in context fusion (ablation) | HPDM · UCF-101, 64×256² | FVD@512: 389.5 -> 378.2 | Helped |
| use 50% width overlap rather than no overlap (ablation) | HPDM tiled inference · UCF-101, 64×256² | FVD@512: 475.05 -> 452.79 | Helped |
| use 50% height overlap rather than no overlap (ablation) | HPDM tiled inference · UCF-101, 64×256² | FVD@512: 475.05 -> 467.36 | Helped |
| use 50% height/width overlap rather than no overlap (ablation) | HPDM tiled inference · UCF-101, 64×256² | FVD@512: 475.05 -> 456.10 | Helped |
| use 50% overlap across frame, height, and width (ablation) | HPDM tiled inference · UCF-101, 64×256² | FVD@512: 475.05 -> 467.37 | No clear effect |
Surprising findings
- Patch-wise training on only ~0.7% of pixels per step still yields consistent full-resolution samples even without full-resolution training or overlapped inference.
- Adaptive computation improves efficiency by about 60% with little performance loss, and slightly improves FVD at 64x256x256 in ablation.
Limitations
- Relies on internal proprietary datasets for text-to-video, limiting reproducibility.
- Text-to-video results are only comparable to foundational models qualitatively and on zero-shot UCF-101; FVD (481.9 at 288x512) is worse than some baselines.
- Dead-pixel artifacts from tokenization/detokenization noted as an open issue.
Experimental setup
Datasets: UCF-101 internal text/video dataset (~25M high-quality + ~70M web videos) internal text/image dataset (~150M)
Benchmarks: UCF-101
Metrics: FVD Inception Score FVD@512 training throughput
Baselines: DIGAN MoCoGAN-HD StyleGAN-V TATS VIDM PVDM Make-A-Video CogVideo PYoCo ImagenVideo
Builds on: Recurrent Interface Networks PatchDiffusion Cascaded diffusion models MultiDiffusion SnapVideo
Closest papers by meaning
- PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution (CVPR 2025 · similarity 0.92)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.90)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.90)
- Patched Denoising Diffusion Models For High-Resolution Image Synthesis (ICLR 2024 · similarity 0.89)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.