Research map / Video / Video generation
SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeedVR, a diffusion transformer with shifted window MMDiT, causal video VAE, and progressive mixed image-video training for generic video restoration at arbitrary length and resolution.
Problem
Diffusion-based video restoration relies on full attention and patch-based tiled sampling, which limits resolution flexibility and makes inference slow on long, high-resolution videos.
Approach
Replaces full attention in MMDiT with large non-overlapping shifted window attention (64x64 over 8x compressed latents) with variable-sized boundary windows and 3D RoPE, paired with a from-scratch causal video VAE (4x temporal, 8x spatial, 16 channels) and progressive mixed image/video training.
Key results
- Achieves best scores on 4 of 6 benchmarks (SPMCS, UDM10, YouHQ40, AIGC38) in Table 1 against diffusion and GAN VSR baselines.
- Causal video VAE reaches rFVD 1.85, about 69.5% lower than the second best (CogVideoX, 6.06).
- SeedVR (2.48B params) is over 2x faster than Upscale-A-Video, VEnhancer, and MGLD-VSR per Figure 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SeedVR instead of Real-ESRGAN | Video restoration · SPMCS | DOVER: 8.566 -> 10.508 | Helped |
| Use SeedVR instead of Upscale-A-Video | Video restoration · UDM10 | DOVER: 9.238 -> 10.537 | Helped |
| Use SeedVR instead of MGLD-VSR | Video restoration · REDS30 | DOVER: 6.544 -> 6.673 | Helped |
| Use SeedVR instead of VEnhancer | Video restoration · YouHQ40 | DOVER: 11.444 -> 12.788 | Helped |
| Use SeedVR instead of VEnhancer | Video restoration · VideoLQ | DOVER: 8.719 -> 8.009 | Hurt |
| Use SeedVR instead of Upscale-A-Video | Video restoration · AIGC38 | DOVER: 12.857 -> 13.424 | Helped |
| Use a 5 × 64 × 64 attention window rather than a 1 × 32 × 32 window (ablation) | SeedVR · YouHQ40 | DOVER: 11.947 -> 11.595 | No clear effect |
| Use a 5 × 64 × 64 attention window rather than a 1 × 64 × 64 window (ablation) | SeedVR · YouHQ40 | DOVER: 10.690 -> 11.595 | Helped |
| Use a 5 × 64 × 64 attention window rather than a 1 × 32 × 32 window (ablation) | SeedVR · YouHQ40 | training time (sec/iter): 58.37 -> 20.29 | Helped |
| Use a 5 × 64 × 64 attention window rather than a 5 × 32 × 32 window (ablation) | SeedVR · YouHQ40 | DOVER: 10.558 -> 11.595 | Helped |
| Use a 5 × 64 × 64 attention window rather than full spatial attention (ablation) | SeedVR · YouHQ40 | DOVER: 8.521 -> 11.595 | Helped |
| Use the SeedVR CV-VAE rather than the CogVideoX VAE | Video autoencoder · video reconstruction | rFVD: 6.06 -> 1.85 | Helped |
Surprising findings
- PSNR and SSIM are lower than several baselines (e.g., RealViFormer, SD x4 Upscaler on some sets), which the authors attribute to focus on perceptual quality.
Limitations
- Sampling efficiency and robustness are still open, as stated by the authors.
- Lower PSNR/SSIM than some baselines on several benchmarks, favoring perceptual metrics.
Experimental setup
Datasets: SPMCS UDM10 REDS30 YouHQ40 VideoLQ AIGC38
Benchmarks: SPMCS UDM10 REDS30 YouHQ40 VideoLQ AIGC38
Metrics: PSNR SSIM LPIPS DISTS NIQE MUSIQ CLIP-IQA DOVER rFVD
Baselines: Real-ESRGAN SD x4 Upscaler ResShift RealViFormer MGLD-VSR Upscale-A-Video VEnhancer
Builds on: MMDiT Swin Transformer Stable Diffusion 3 NaViT Upscale-A-Video
Closest papers by meaning
- SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Training (ICLR 2026 · similarity 0.92)
- AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration (ICML 2025 · similarity 0.88)
- DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution (NeurIPS 2025 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.86)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.86)
- Zero-shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.