Research map / Video / Video segmentation and tracking
Space-time Mixing Attention for Video Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add space-time mixing attention with window tw=1 (vs spatial-only tw=0) (ablation) | X-ViT ViT-B/16 temporal avg · SSv2 | top-1 accuracy: 45.2 -> 62.5 | Helped |
| increase local window from tw=1 to tw=2 (ablation) | X-ViT · SSv2 | top-1 accuracy: 62.5 -> 60.5 | Hurt |
| apply space-time mixing to all layers instead of half (ablation) | X-ViT · SSv2 | top-1 accuracy: 61.7 -> 62.6 | Helped |
| replace temporal averaging with one Temporal Attention layer (ablation) | X-ViT · SSv2 | top-1 accuracy: 62.4 -> 64.4 | Helped |
| increase TA layers from 1 to 3 (ablation) | X-ViT · SSv2 | top-1 accuracy: 64.4 -> 64.5 | No clear effect |
| add space-time mixing to spatial-only attention with TA (ablation) | X-ViT · SSv2 | top-1 accuracy: 56.6 -> 64.4 | Helped |
| mix key and value instead of input token or value only (ablation) | X-ViT · SSv2 | top-1 accuracy: 63.1 -> 64.4 | Helped |
| mix 100% of channels instead of 50% (ablation) | X-ViT · SSv2 | top-1 accuracy: 64.4 -> 62.5 | Hurt |
| use summary token instead of TA (ablation) | X-ViT · SSv2 | top-1 accuracy: 64.4 -> 63.7 | Hurt |
| use summary token instead of TA (ablation) | X-ViT · Kinetics-400 | top-1 accuracy: 78.5 -> 78.7 | Helped |
| use ViT-B/16 instead of ViT-B/32 (more tokens) (ablation) | X-ViT · SSv2 | top-1 accuracy: 60.5 -> 64.4 | Helped |
| increase frames from 8 to 16 | X-ViT · SSv2 | top-1 accuracy: 64.4 -> 66.2 | Helped |
Closest papers by meaning
- Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (NeurIPS 2021 · similarity 0.88)
- Is Space-Time Attention All You Need for Video Understanding? (ICML 2021 · similarity 0.88)
- Video Swin Transformer (CVPR 2022 · similarity 0.88)
- Multiview Transformers for Video Recognition (CVPR 2022 · similarity 0.87)
- Video Frame Interpolation Transformer (CVPR 2022 · similarity 0.87)
- TGBFormer: Transformer-GraphFormer Blender Network for Video Object Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.