Research map / Video / Video segmentation and tracking
Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMA, a data sampling method that builds a scale pyramid of grid-based fragments and masks it into a regular input size so a single-branch Swin/VideoSwin model captures both local and global quality cues for IQA and VQA.
Problem
Image and video quality assessment needs both local detail and global semantics, but standard sampling (resizing, cropping, grid fragments) captures only one, and existing fixes (multi-branch or multi-scale inputs) increase model complexity.
Approach
Build a multi-scale pyramid of the input, sample grid-based fragments from each scale, and interlace the scales with spatial or temporal masks so the result has the same size as a single-scale fragment input, allowing a single-branch model with no extra complexity.
Key results
- On LSVQ-trained VQA, SAMA reaches SRCC 0.883 on LSVQ versus 0.870 for the reproduced FAST-VQA baseline, and 0.880 vs 0.860 on KoNViD.
- Intra-dataset VQA on YouTube-UGC improves SRCC from 0.842 (baseline) to 0.881 (SAMA).
- On KonIQ, SAMA reaches SRCC 0.930 vs 0.923 for the baseline; on SPAQ it reaches 0.925 vs 0.908.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace FAST-VQA fragment sampling with SAMA progressive temporal masking | FAST-VQA · LSVQ | SRCC: 0.870 -> 0.883 | Helped |
| Replace FAST-VQA fragment sampling with SAMA progressive temporal masking | FAST-VQA · LSVQ-1080p | SRCC: 0.766 -> 0.782 | Helped |
| Replace FAST-VQA fragment sampling with SAMA progressive temporal masking | FAST-VQA · KoNViD (trained on LSVQ) | SRCC: 0.860 -> 0.880 | Helped |
| Replace FAST-VQA fragment sampling with SAMA progressive temporal masking | FAST-VQA · LIVE-VQC (trained on LSVQ) | SRCC: 0.816 -> 0.834 | Helped |
| Replace SwinTransformer baseline sampling with SAMA window-based spatial masking | SwinTransformer IQA baseline · KonIQ | SRCC: 0.923 -> 0.930 | Helped |
| Replace SwinTransformer baseline sampling with SAMA window-based spatial masking | SwinTransformer IQA baseline · SPAQ | SRCC: 0.908 -> 0.925 | Helped |
| Use choppy temporal masking instead of progressive masking (ablation) | SAMA · LSVQ | SRCC: 0.883 -> 0.879 | No clear effect |
| Use mixed choppy and progressive temporal masking (ablation) | SAMA · LSVQ | SRCC: 0.883 -> 0.880 | No clear effect |
| Add spatial patch-based masking (ablation) | SAMA · LSVQ | SRCC: 0.883 -> 0.878 | Hurt |
| Add spatial window-based masking (ablation) | SAMA · LSVQ | SRCC: 0.883 -> 0.877 | Hurt |
| Use spatial patch-based mask instead of spatial window-based mask (ablation) | SAMA · KonIQ | SRCC: 0.930 -> 0.926 | Hurt |
| Add learned temporal weights for relative scale encoding (ablation) | SAMA · LSVQ | SRCC: 0.883 -> 0.882 | No clear effect |
Surprising findings
- Relative scale encoding modules (SAMA-W, SAMA-SE, relative scale bias) give little or no improvement over plain SAMA, suggesting the scale-interlaced input is already recognizable by the backbone.
- Mixing spatial and temporal masks decreases performance on VQA, which the authors attribute to harder learning on the small-scale dataset.
Limitations
- Evaluated only with a Swin/VideoSwin single-branch backbone; generalization to other transformer architectures is untested.
- Multi-branch baselines were weakened by the authors, so the comparison with them is not fully like-for-like.
- Mask designs are hand-chosen patterns (window, patch, choppy, progressive, mixed) rather than learned or systematically searched.
Experimental setup
Datasets: LSVQ KoNViD LIVE-VQC YouTube-UGC KonIQ SPAQ
Benchmarks: LSVQ LSVQ-1080p KoNViD LIVE-VQC YouTube-UGC KonIQ SPAQ
Metrics: SRCC PLCC
Baselines: FAST-VQA VSFA TLVQM VIDEVAL PVQ BVQA-2022 DSD-PRO ZoomVQA DOVER HyperIQA MUSIQ IEIT
Builds on: FAST-VQA Swin Transformer VideoSwin MUSIQ masked autoencoder
Closest papers by meaning
- IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis (ICLR 2025 · similarity 0.86)
- Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare (NeurIPS 2024 · similarity 0.85)
- LAVIB: A Large-scale Video Interpolation Benchmark (NeurIPS 2024 · similarity 0.85)
- Toward Generalized Image Quality Assessment: Relaxing the Perfect Reference Quality Assumption (CVPR 2025 · similarity 0.85)
- KVQ: Boosting Video Quality Assessment via Saliency-guided Local Perception (CVPR 2025 · similarity 0.85)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.