Research map / Video / Action recognition
Is Space-Time Attention All You Need for Video Understanding?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use divided space-time attention instead of joint space-time attention (ablation) | TimeSformer · Kinetics-400 | Video-level accuracy: 77.4 -> 78.0 | Helped |
| Use divided space-time attention instead of joint space-time attention (ablation) | TimeSformer · Something-Something-V2 | Video-level accuracy: 58.5 -> 59.5 | Helped |
| Add temporal attention to space-only attention (ablation) | TimeSformer · Something-Something-V2 | Video-level accuracy: 36.6 -> 59.5 | Helped |
| Use space-time rather than space-only positional embeddings (ablation) | TimeSformer · Kinetics-400 | Video-level accuracy: 77.8 -> 78.0 | Helped |
| Use space-time rather than space-only positional embeddings (ablation) | TimeSformer · Something-Something-V2 | Video-level accuracy: 52.5 -> 59.5 | Helped |
| Remove positional embeddings (ablation) | TimeSformer · Kinetics-400 | Video-level accuracy: 78.0 -> 75.4 | Hurt |
| Remove positional embeddings (ablation) | TimeSformer · Something-Something-V2 | Video-level accuracy: 59.5 -> 45.8 | Hurt |
| Increase model size from ViT-Base to ViT-Large (ablation) | TimeSformer · Kinetics-400 and Something-Something-V2 | Accuracy: 1% worse | Hurt |
| Decrease model size from ViT-Base to ViT-Small (ablation) | TimeSformer · Kinetics-400 and Something-Something-V2 | Accuracy: about 5% worse | Hurt |
| Increase patch size from 16 to 32 (ablation) | TimeSformer · Kinetics-400 and Something-Something-V2 | Accuracy: about 3% worse | Hurt |
| Apply spatial attention before temporal attention (ablation) | TimeSformer divided attention · Kinetics-400 and Something-Something-V2 | Accuracy: 0.5% drop | Hurt |
| Apply spatial and temporal attention in parallel instead of sequentially (ablation) | TimeSformer · Kinetics-400 and Something-Something-V2 | Accuracy: 0.4% lower | Hurt |
Closest papers by meaning
- Space-time Mixing Attention for Video Transformer (NeurIPS 2021 · similarity 0.88)
- Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification (CVPR 2018 · similarity 0.87)
- Relational Self-Attention: What's Missing in Attention for Video Understanding (NeurIPS 2021 · similarity 0.87)
- Time Is MattEr: Temporal Self-supervision for Video Transformers (ICML 2022 · similarity 0.86)
- Shrinking Temporal Attention in Transformers for Video Action Recognition (AAAI 2022 · similarity 0.86)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.