Research map / Video / Video segmentation and tracking
Efficient Video Transformers with Spatial-Temporal Token Selection
Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wu, Yu-Gang Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use temporal rather than spatial selection at block 0 (same ~50% FLOPs cut) (ablation) | STTS on MViT-B16 · Kinetics-400 (MViT-B16, ~50% GFLOPs reduction) | top-1 accuracy: 76.2 -> 77.5 | Improved |
| add joint temporal (block 0) and spatial (block 4) selection instead of single-dimension (ablation) | STTS on MViT-B16 · Kinetics-400 (MViT-B16, ~50% GFLOPs reduction) | top-1 accuracy: 77.3 -> 77.5 | Improved |
| replace single-step with multi-step selection (0, 4, 8) (ablation) | STTS on MViT-B16 · Kinetics-400 (MViT-B16, ~50% GFLOPs reduction) | top-1 accuracy: 77.5 -> 76.6 | Hurt |
| use anchor-based instead of token-based spatial selection (ablation) | STTS on MViT-B16-S0_0.5 · Kinetics-400 (MViT-B16, anchor-based spatial selection ablation) | top-1 accuracy: 57.6 -> 76.2 | Improved |
| use anchors for random selection (ablation) | Rand.-S0_0.5 · Kinetics-400 (MViT-B16, anchor-based spatial selection ablation) | top-1 accuracy: 16.7 -> 75.6 | Improved |
| score on token-level rather than anchor-level features (ablation) | STTS on MViT-B16-S0_0.5 · Kinetics-400 (MViT-B16, anchor-based spatial selection ablation) | top-1 accuracy: 75.4 -> 76.2 | Improved |
| vary sigma 0.05 / 0.1 / 0.2 in perturbed Top-K (ablation) | MViT-B16-T0_0.5 · Kinetics-400 (MViT-B16, sigma sweep) | top-1 accuracy: 77.2, 77.3, 77.1 | No clear effect |
| learned differentiable Top-K selection instead of random selection (ablation) | MViT-B16-T0_0.5 · Kinetics-400 (MViT-B16, sigma sweep) | top-1 accuracy: 75.5 -> 77.3 | Improved |
| add STTS temporal selection at 0.6 ratio to VideoSwin-B (ablation) | VideoSwin-B · Kinetics-400 | top-1 accuracy: 82.7 -> 81.4 | Hurt |
| add STTS temporal selection at 0.7 ratio to VideoSwin-B (ablation) | VideoSwin-B · Something-Something-V2 | top-1 accuracy: 69.6 -> 68.7 | Hurt |
| add STTS (T0.9 S4.9) to MViT-B16 for less compute (ablation) | MViT-B16 · Kinetics-400 | top-1 accuracy: 78.4 -> 78.1 | No clear effect |
| add STTS T0.6-S4.9 to MViT-B16 for speed | MViT-B16 · Kinetics-400 (MViT-B16, inference speed) | throughput (videos/s): 79.3 -> 129.7 | Improved |
Closest papers by meaning
- Token Selection is a Simple Booster for Vision Transformers (TPAMI 2022 · similarity 0.88)
- Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers (NeurIPS 2026 · similarity 0.87)
- Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (NeurIPS 2024 · similarity 0.87)
- vid-TLDR: Training Free Token Merging for Light-weight Video Transformer (CVPR 2024 · similarity 0.86)
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.86)
- LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
