Research map / Video / Video segmentation and tracking
Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SATeCo, which adapts a frozen Stable Diffusion UNet and VAE decoder for video super-resolution using spatial feature adaptation (pixel-wise affine modulation) and temporal feature alignment (tubelet self- and cross-attention) modules.
Problem
Video super-resolution with diffusion models must preserve visual fidelity against diffusion stochasticity and keep temporal consistency across HR frames.
Approach
Freezes pre-trained UNet and VAE and trains only SFA modules (per-pixel affine modulation predicted from LR latent features) and TFA modules (3D-tubelet self-attention plus cross-attention to LR tubelets) inserted into decoder blocks, plus a transformer video upscaler and a trainable video refiner.
Key results
- On REDS4, SATeCo reaches 31.62 dB PSNR, close to IconVSR's 31.60 dB, and achieves the best LPIPS (0.1735), DISTS (0.0607), NIQE (4.104 close to best 4.104 of IconVSR tie) and CLIP-IQA (0.6622) among compared methods.
- On Vid4, SATeCo attains DISTS of 0.1015, a 26.0% relative reduction over the best competitor VRT.
- In a MTurk study with 100 evaluators, SATeCo was preferred over VRT, BasicVSR, IconVSR and StableSR in head-to-head comparisons (preference ratios 66% to 81%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SATeCo instead of IconVSR | Video super-resolution · REDS4 | LPIPS: 0.1939 -> 0.1735 | Helped |
| Use SATeCo instead of VRT | Video super-resolution · Vid4 | DISTS: 0.1372 -> 0.1015 | Helped |
| Add SFA to the UNet (ablation) | Diffusion model for video super-resolution · REDS4 | PSNR: 28.56 -> 28.93 | Helped |
| Add TFA alongside SFA to the UNet (ablation) | Diffusion model for video super-resolution · REDS4 | PSNR: 28.93 -> 29.45 | Helped |
| Add SFA and TFA to the VAE decoder (ablation) | Model C · REDS4 | PSNR: 29.45 -> 31.62 | Helped |
| Replace the transformer-based video upscaler with PixelShuffle (ablation) | SATeCo · REDS4 | PSNR: 31.62 -> 29.77 | Hurt |
| Set video refiner w to 0 (ablation) | SATeCo · REDS4 | PSNR: 31.62 -> 30.36 | Hurt |
| Set video refiner w to 1.0 (ablation) | SATeCo · REDS4 | PSNR: 31.62 -> 28.99 | Hurt |
| Set video refiner w to 0 instead of 0.5 (ablation) | SATeCo · REDS4 | LPIPS: 0.1735 -> 0.1581 | Helped |
| Set video refiner w to 0 instead of 0.5 (ablation) | SATeCo · REDS4 | DISTS: 0.0607 -> 0.0339 | Helped |
| Set video refiner w to 0 instead of 0.5 (ablation) | SATeCo · REDS4 | NIQE: 4.104 -> 3.457 | Helped |
Surprising findings
- Setting the refiner trade-off w=0 gives the best perception metrics (DISTS 0.0339) but lower PSNR, showing a fidelity-perception trade-off.
Limitations
- Pixel-based PSNR/SSIM remain below some regression methods on Vid4 (e.g., VRT and IconVSR are higher).
- Evaluated only on REDS4 and Vid4 benchmarks.
- Multi-stage training with separate upscaler, UNet, VAE, and refiner stages.
Experimental setup
Datasets: Vimeo-90K
Benchmarks: REDS4 Vid4
Metrics: PSNR SSIM LPIPS DISTS NIQE CLIP-IQA user preference
Baselines: Bicubic StableSR TOFlow EDVR-M BasicVSR VRT IconVSR
Builds on: Stable Diffusion VRT StableSR ControlNet
Closest papers by meaning
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.89)
- LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution (ICML 2026 · similarity 0.88)
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (CVPR 2024 · similarity 0.88)
- FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation (CVPR 2024 · similarity 0.87)
- SkipSR: Faster Super-Resolution with Token Skipping (NeurIPS 2026 · similarity 0.86)
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.