Research map / Video / Video segmentation and tracking
Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIA-VSR, a recurrent Transformer for video super-resolution that uses an intra/inter-frame attention block and a learned feature-level block-wise mask to skip redundant computation between adjacent frames.
Problem
Transformer-based video super-resolution is accurate but has heavy computation and large memory footprint, which hinders deployment on constrained devices; whether skip-style temporal redundancy exploitation works for low-level VSR is unclear.
Approach
Only current-frame features generate attention queries while previously enhanced features supply keys and values (intra/inter-frame attention), and a Gumbel-softmax-trained mask predictor on normalized feature differences skips block-wise computation in attention and FFN layers.
Key results
- On REDS4 with 16/14-frame training, MIA-VSR reaches 32.78 dB PSNR versus 32.72 dB for PSRT-recurrent, with 1.61T FLOPs versus 2.39T.
- With 6-frame training, MIA-VSR achieves 32.01 dB on REDS4 versus 31.88 dB for PSRT-recurrent.
- Peak memory on REDS is 206M versus 2149M for VRT, though runtime (822 ms) is slower than RVRT (473 ms) which uses custom CUDA kernels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace multi-frame self-attention with intra/inter-frame attention (ablation) | VSR model · REDS4 | PSNR: 31.03 -> 31.12 | Helped |
| replace multi-frame self-attention with intra/inter-frame attention (ablation) | VSR model · REDS4 | FLOPs: 871.59 -> 518.92 | Helped |
| add a handcrafted mask with threshold 0.2 (ablation) | IIAB-VSR · REDS4 | PSNR: 31.12 -> 29.83 | Hurt |
| add adaptive mask prediction with ¼=1e-4 (ablation) | IIAB-VSR · REDS4 | FLOPs: 518.92 -> 506.28 | Helped |
| increase mask-loss weight to ¼=3e-4 (ablation) | MIA-VSR · REDS4 | FLOPs: 506.28 -> 469.78 | Helped |
| increase mask-loss weight to ¼=5e-4 (ablation) | MIA-VSR · REDS4 | PSNR: 31.07 -> 31.01 | Hurt |
| increase mask-loss weight to ¼=5e-4 (ablation) | MIA-VSR · REDS4 | FLOPs: 469.78 -> 442.32 | Helped |
| increase mask-loss weight to ¼=1e-3 (ablation) | MIA-VSR · REDS4 | PSNR: 31.01 -> 30.76 | Hurt |
| increase mask-loss weight to ¼=1e-3 (ablation) | MIA-VSR · REDS4 | FLOPs: 442.32 -> 426.84 | Helped |
| use MIA-VSR instead of PSRT-recurrent | 4× VSR · REDS4 | PSNR: 32.72 -> 32.78 | Helped |
| use MIA-VSR instead of PSRT-recurrent | 4× VSR · Vimeo-90K-T | PSNR: 38.27 -> 38.22 | Hurt |
| use MIA-VSR instead of PSRT-recurrent | 4× VSR · Vid4 | PSNR: 28.07 -> 28.20 | Helped |
Surprising findings
- A uniform handcrafted threshold on feature differences for masking drops REDS4 PSNR from 31.12 to 29.83 dB, showing learned adaptive masks are necessary.
Limitations
- Runtime is not faster than RVRT, which uses customized CUDA kernels.
- Masking ratio is controlled by a hyperparameter trading accuracy against computation.
- Evaluated only on REDS, Vimeo-90K and Vid4 for 4x super-resolution.
Experimental setup
Datasets: REDS Vimeo-90K
Benchmarks: REDS4 Vimeo-90K-T Vid4
Metrics: PSNR SSIM FLOPs parameters peak memory runtime
Baselines: EDVR MuCAN VSR-T VRT RVRT PSRT-recurrent BasicVSR BasicVSR++ TTVSR TOFlow
Builds on: BasicVSR++ PSRT Swin Transformer skip-convolution
Closest papers by meaning
- Activating More Pixels in Image Super-Resolution Transformer (CVPR 2023 · similarity 0.85)
- Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary (CVPR 2024 · similarity 0.83)
- Efficient Attention-Sharing Information Distillation Transformer for Lightweight Single Image Super-Resolution (AAAI 2025 · similarity 0.83)
- Recursive Generalization Transformer for Image Super-Resolution (ICLR 2024 · similarity 0.83)
- CFAT: Unleashing Triangular Windows for Image Super-resolution (CVPR 2024 · similarity 0.82)
- Transformer-Based Selective Super-resolution for Efficient Image Refinement (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.