Research map / Video / Video segmentation and tracking
Semantic Lens: Instance-Centric Semantic Alignment for Video Super-resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semantic Lens, a video super-resolution paradigm that extracts instance, event, and scene semantics with X-Decoder and injects them into a BasicVSR++-style pixel enhancer via a SPACE block (GPS + ISEE) and an IMAGE pre-alignment module.
Problem
Accurate pixel-level inter-frame alignment in video super-resolution is hard due to intricate, mixed motions (camera motion, changing scenes, inconsistent instance trajectories, occlusion).
Approach
Model video as instance-centric semantics (global scene tokens, instance tokens across frames from a VIS-style extractor) and use them to modulate pixel features (affine GPS) and drive cross-attention alignment within the same instance, plus instance-mask-masked coarse pre-alignment.
Key results
- On 4x BI degradation on YTVIS-19, Semantic Lens reaches 37.19 dB / 0.9542 vs. PSRT 36.58 dB / 0.9514.
- Under BD degradation on YTVIS-19, Semantic Lens reaches 38.49 dB vs. 34.95 dB for PSRT (the next best).
- Ablation on YTVIS-19 shows GPS, ISEE, and IMAGE each add roughly 0.05 to 0.3 dB PSNR over the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Semantic Lens instead of PSRT | VSR · YTVIS-19 BI | PSNR: 36.58 -> 37.19 | Helped |
| use Semantic Lens instead of IART | VSR · YTVIS-21 BI | PSNR: 36.24 -> 36.55 | Helped |
| use Semantic Lens instead of IART | VSR · YTVIS-22 BI | PSNR: 36.12 -> 36.39 | Helped |
| use Semantic Lens instead of PSRT | VSR · YTVIS-19 BD | PSNR: 34.95 -> 38.49 | Helped |
| use Semantic Lens instead of PSRT | VSR · YTVIS-21 BD | PSNR: 32.07 -> 37.93 | Helped |
| use Semantic Lens instead of PSRT | VSR · YTVIS-22 BD | PSNR: 31.98 -> 37.73 | Helped |
| add GPS (ablation) | Semantic Lens baseline (MFSAB only) · YTVIS-19 | PSNR: 36.75 -> 36.84 | Helped |
| add ISEE (ablation) | Baseline + GPS · YTVIS-19 | PSNR: 36.84 -> 37.14 | Helped |
| add IMAGE (ablation) | Baseline + GPS + ISEE · YTVIS-19 | PSNR: 37.14 -> 37.19 | Helped |
Surprising findings
- Under BD degradation, the gain over prior methods is much larger (about 3.5 dB over PSRT on YTVIS-19) than under BI degradation (about 0.6 dB), which the paper does not explain in depth.
Limitations
- Relies on a pretrained video instance segmentation extractor (X-Decoder) trained on YouTube-VIS-style data, so it depends on the quality of semantic priors.
- Evaluated only on YouTube-VIS videos with synthetic degradations; no real-world degradation evaluation is reported.
- Gains over baselines are reported on a single dataset family; the paper does not analyze computational cost.
Experimental setup
Datasets: YouTube-VIS 2019 YouTube-VIS 2021 YouTube-VIS 2022
Benchmarks: YTVIS-19 YTVIS-21 YTVIS-22
Metrics: PSNR SSIM
Baselines: EDVR BasicVSR IconVSR BasicVSR++ TTVSR RVRT PSRT IART
Builds on: BasicVSR++ X-Decoder DCN v2 MFSAB (multi-frame self-attention block from PSRT)
Closest papers by meaning
- SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution (NeurIPS 2024 · similarity 0.90)
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.87)
- SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning (NeurIPS 2025 · similarity 0.86)
- Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach (CVPR 2025 · similarity 0.86)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.86)
- SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.