Research map / Video / Video segmentation and tracking
CSTA: CNN-based Spatiotemporal Attention for Video Summarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSTA, which stacks frame features into an image-like 3-channel representation and applies a GoogleNet 2D CNN as a sliding-window attention module for supervised video summarization on SumMe and TVSum with fewer MACs.
Problem
Attention-based video summarization captures temporal dependencies but fails to model visual (spatial) significance within frames, and prior spatiotemporal fixes add costly extra modules.
Approach
Treat the stacked frame-feature matrix (with a CLS token) as an image and apply a 2D CNN to produce attention maps that capture both temporal and visual importance via absolute-position cues, combining temporal (softmax over time) and visual (softmax over dimensions) attention with features in a single lightweight pass.
Key results
- CSTA achieves Kendall's tau 0.246 and Spearman's rho 0.274 on SumMe, the best average rank among compared methods.
- On TVSum CSTA reaches tau 0.194 and rho 0.255, second-best average rank behind DMASum (tau 0.203).
- CSTA requires 9.78G MACs for score prediction on SumMe versus 1.43G for VASNet and 3.63G for MSVA, with feature extraction of 413.03G shared with other CNN-based methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the CNN Attention Module (ablation) | CSTA · SumMe | Spearman's ρ: 0.197 -> 0.205 | Helped |
| Add the CNN Attention Module (ablation) | CSTA · TVSum | Spearman's ρ: 0.163 -> 0.231 | Helped |
| Add dimension-axis (visual) attention (ablation) | CSTA · SumMe | Spearman's ρ: 0.205 -> 0.211 | Helped |
| Add dimension-axis (visual) attention (ablation) | CSTA · TVSum | Spearman's ρ: 0.231 -> 0.240 | Helped |
| Add key and value linear projections (ablation) | CSTA · SumMe | Spearman's ρ: 0.211 -> 0.231 | Helped |
| Add key and value linear projections (ablation) | CSTA · TVSum | Spearman's ρ: 0.240 -> 0.253 | Helped |
| Add positional encoding to attention maps (ablation) | CSTA · SumMe | Spearman's ρ: 0.231 -> 0.251 | Helped |
| Add positional encoding to attention maps (ablation) | CSTA · TVSum | Kendall's τ: 0.193 -> 0.189 | Hurt |
| Add positional encoding to attention maps (ablation) | CSTA · TVSum | Spearman's ρ: 0.253 -> 0.248 | Hurt |
| Add the CLS token (ablation) | CSTA · SumMe | Spearman's ρ: 0.251 -> 0.257 | Helped |
| Add the CLS token (ablation) | CSTA · TVSum | Spearman's ρ: 0.248 -> 0.254 | Helped |
| Add the skip connection (ablation) | CSTA · SumMe | Spearman's ρ: 0.257 -> 0.274 | Helped |
Surprising findings
- Plain CNN backbones (MobileNet, EfficientNet, GoogleNet, ResNet) applied directly to frame features outperform many prior video summarization models on SumMe.
- The authors argue F1 score is misleading for video summarization because it favors selecting many short shots, and so they rely on rank correlation metrics.
Limitations
- Relies on the F1-free rank correlation evaluation; F1 comparisons with prior work are not reported.
- Performance on TVSum is second-best rather than best, behind DMASum.
- Uses frozen GoogleNet features and an offline supervised setup evaluated only on two small benchmarks.
Experimental setup
Datasets: SumMe TVSum
Benchmarks: SumMe TVSum
Metrics: Kendall's tau Spearman's rho MACs rank
Baselines: dppLSTM DAC HSA-RNN DAN STVT DSNet-AF DSNet-AB HMT VJMHT CLIP-It iPTNet A2Summ
Builds on: STVT PGL-SUM VASNet GoogleNet CPVT
Closest papers by meaning
- Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer (CVPR 2022 · similarity 0.87)
- Motion Guided Spatial Attention for Video Captioning (AAAI 2019 · similarity 0.87)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.85)
- Compositional Video Understanding with Spatiotemporal Structure-based Transformers (CVPR 2024 · similarity 0.85)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.84)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.