Research map / Video / Video segmentation and tracking
Self-supervised ControlNet with Spatio-Temporal Mamba for Real-world Video Super-resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCST, a real-world video super-resolution framework that adds a self-supervised (MoCo-style) ControlNet and a Spatio-Temporal Continuous Mamba block to a pretrained Stable Diffusion model, trained in three stages on REDS and YouHQ.
Problem
Diffusion-based video super-resolution suffers from temporal inconsistency due to sampling randomness, and from artifacts when unknown complex degradations in LR videos disturb conditioning.
Approach
Combines a momentum-contrastive (patch-level MoCo) ControlNet that aligns LR features to HR features to extract degradation-insensitive conditioning, with a 3D Mamba block using a continuous spatial-temporal scan (K=6 paths) for global spatio-temporal attention at linear cost, plus a three-stage HR-LR mixed training schedule.
Key results
- On YouHQ40 SCST reaches 24.31 PSNR and 0.2525 LPIPS, versus 24.44 PSNR for RealViformer, but with better perceptual scores (LPIPS and DISTS best across all synthetic sets).
- On real-world VideoLQ, SCST attains the best CLIP-IQA (0.4859), MUSIQ (59.20), NIQE (3.566) and DOVER (0.7443).
- Ablation on YouHQ: adding MoCoCtrl to the baseline raises PSNR from 21.22 to 23.63; adding STCM on top raises it to 24.31 with LPIPS 0.2525.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoCoCtrl contrastive learning (ablation) | SCST · YouHQ | PSNR: 23.63 dB vs. 24.31 dB | Helped |
| Add MoCoCtrl contrastive learning (ablation) | SCST · YouHQ | LPIPS: 0.2671 -> 0.2525 | Helped |
| Add MoCoCtrl contrastive learning (ablation) | SCST · YouHQ | SSIM: 0.6563 -> 0.6758 | Helped |
| Add MoCoCtrl contrastive learning (ablation) | SCST · YouHQ | DISTS: 0.1473 -> 0.1344 | Helped |
| Add STCM (ablation) | SCST with MoCoCtrl · YouHQ | PSNR: 23.18 -> 24.31 | Helped |
| Add STCM (ablation) | SCST with MoCoCtrl · YouHQ | LPIPS: 0.2581 -> 0.2525 | Helped |
| Use STCM instead of no temporal module (ablation) | SCST · YouHQ | Warp Error: 0.3603 -> 0.2295 | Helped |
| Use STCM instead of Local Attention (ablation) | SCST · REDS4 | PSNR: 22.60 -> 23.02 | Helped |
| Use STCM instead of Mamba-S 3D Sweep Scan (ablation) | SCST · REDS4 | PSNR: 22.59 -> 23.02 | Helped |
| Use STCM instead of Mamba-S 3D Sweep Scan (ablation) | SCST · YouHQ | Warp Error: 0.2494 -> 0.2295 | Helped |
| Use STCM instead of Local Attention (ablation) | SCST · YouHQ | PSNR: 24.07 -> 24.31 | Helped |
| Use STCM instead of Local Attention (ablation) | SCST · YouHQ | Warp Error: 0.2311 -> 0.2295 | Helped |
Surprising findings
- SCST does not win PSNR/SSIM on every synthetic dataset; RealViformer and DBVSR score higher PSNR on some sets, which the authors attribute to blurrier outputs.
Limitations
- Inference requires segmenting long videos into sequences due to memory limits.
- Gains in PSNR/SSIM are not consistent across synthetic datasets.
Experimental setup
Datasets: REDS YouHQ UDM10 SPMCS VideoLQ
Benchmarks: REDS4 UDM10 SPMCS YouHQ40 VideoLQ
Metrics: PSNR SSIM LPIPS DISTS CLIP-IQA MUSIQ NIQE DOVER Warping Error
Baselines: RealESRGAN StableSR DBVSR RealBasicVSR RealViformer Upscale-A-Video MGLD
Builds on: Stable Diffusion ControlNet MoCo Mamba RealBasicVSR
Closest papers by meaning
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.88)
- Event-based Video Super-Resolution via State Space Models (CVPR 2025 · similarity 0.87)
- DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution (NeurIPS 2025 · similarity 0.87)
- SAVSR: Arbitrary-Scale Video Super-Resolution via a Learned Scale-Adaptive Network (AAAI 2024 · similarity 0.87)
- Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning (CVPR 2024 · similarity 0.87)
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.