Research map / Video / Video segmentation and tracking
Enhancing Video Super-Resolution via Implicit Resampling-based Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that resampling choice matters for frame alignment in video super-resolution and proposes implicit resampling-based alignment (IA) using a positional-encoded coordinate network and window-based cross-attention, plugged into BasicVSR and PSRT-recurrent.
Problem
Frame-wise alignment in video super-resolution relies on resampling sub-pixel values, but existing methods default to bilinear interpolation, which low-pass filters the signal and hinders high-frequency reconstruction; nearest-neighbour introduces spatial distortions.
Approach
Replace explicit interpolation with an implicit resampling: decimal flow offsets are encoded into sinusoidal positional encodings, and the aligned value is estimated by cross-attention over a flow-centred window of reference features, with no smoothness constraint imposed.
Key results
- On the Sintel alignment study with GT flow, implicit alignment reaches 32.14 PSNR versus 31.92 for bilinear warping and 32.08 for FGDC.
- IA-RT reaches 32.90 PSNR / 0.9138 SSIM on REDS4 (BI), above PSRT-recurrent at 32.72 PSNR.
- IA-CNN improves BasicVSR on REDS4 from 31.42 to 31.68 PSNR with 8.5M vs 6.3M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · REDS4, BI degradation | PSNR: 32.72 -> 32.90 | Helped |
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · Vimeo-90K-T, BI degradation | PSNR: 38.27 -> 38.14 | Hurt |
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · Vid4, BI degradation | PSNR: 28.07 -> 28.26 | Helped |
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · UDM10, BD degradation | PSNR: 41.15 | Helped |
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · Vimeo-90K-T, BD degradation | PSNR: 38.72 -> 38.62 | Hurt |
| replace PSRT-recurrent alignment with implicit resampling alignment | PSRT-recurrent (IA-RT) · Vid4, BD degradation | PSNR: 29.54 -> 29.68 | Helped |
| add positional encoding on decimal offsets and window indices (ablation) | IA · Sintel | PSNR: 30.43 -> 30.71 | Helped |
| add positional encoding only on window indices (ablation) | IA · Sintel | PSNR: 30.43 -> 30.54 | No clear effect |
| add positional encoding only on decimal offsets (ablation) | IA · Sintel | PSNR: 30.43 -> 28.71 | Hurt |
| vary cross-attention window size from 2x2 to 3x3 or 4x4 (ablation) | IA · Sintel, GT Flow | PSNR: 32.06; 32.06; 32.05 | No clear effect |
| vary cross-attention window size from 2x2 to 3x3 or 4x4 (ablation) | IA · Sintel, SpyNet Flow | PSNR: 31.97; 31.95; 31.96 | No clear effect |
Surprising findings
- Nearest-neighbour resampling outperforms bilinear for image alignment, while bilinear is better for feature alignment.
- IA-RT is slightly below PSRT and VRT on Vimeo-90K-T.
Limitations
- Implicit resampling reduces interpretability of the alignment.
- Gains are limited when optical flow is inaccurate, e.g. IA-RT is slightly below PSRT and VRT on Vimeo-90K-T.
- Larger windows add robustness but reduce alignment sharpness, so window size needs tuning.
Experimental setup
Datasets: Sintel REDS Vimeo-90K Vid4 UDM10 VideoLQ
Benchmarks: Sintel REDS4 Vimeo-90K-T Vid4 UDM10 VideoLQ
Metrics: PSNR SSIM parameters FLOPs runtime
Baselines: BasicVSR BasicVSR++ VRT RVRT PSRT-recurrent EDVR FGDC FGDA patch alignment bilinear warping nearest-neighbour warping bicubic warping
Builds on: BasicVSR PSRT deformable convolution alignment coordinate networks flow-guided deformable attention
Closest papers by meaning
- Arbitrary-Scale Video Super-resolution Guided by Dynamic Context (AAAI 2024 · similarity 0.87)
- BasicVSR++: Improving Video Super-Resolution With Enhanced Propagation and Alignment (CVPR 2022 · similarity 0.87)
- BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (CVPR 2021 · similarity 0.87)
- Sequence Matters: Harnessing Video Models in 3D Super-Resolution (AAAI 2025 · similarity 0.86)
- VideoGigaGAN: Towards Detail-rich Video Super-Resolution (CVPR 2025 · similarity 0.86)
- BF-STVSR: B-Splines and Fourier---Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.