Research map / Architectures and efficiency / Attention and transformers
Don't Look Twice: Faster Video Transformers with Run-Length Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Run-Length Tokenization (RLT), which removes temporally repeated video patches before the transformer and adds a learnable run-length positional encoding, to speed up video transformer training and inference with little accuracy loss.
Problem
Video transformers are slow to train because uniform spatiotemporal patch tokenization yields huge token counts, many of which are redundant over time; existing pruning/merging/masking methods either add overhead, need padding, or are not content-aware.
Approach
Compare temporally consecutive raw patches (not embeddings) with an L1 threshold, drop static repeats before inference, and encode each surviving token's run length with a learnable length positional encoding; packs variable-length examples with block-diagonal attention so standard fast attention kernels still apply.
Key results
- On Kinetics-400, RLT fine-tuning of ViT-L matches baseline accuracy (84.7 vs 84.8) while reducing fine-tuning time from 21.6h to 15.4h (1.4x) on 8 GPUs.
- Inference-only RLT with tau=0.1 on ViT-H K400 gives 86.3 accuracy vs 86.8 baseline at 1.45x throughput.
- On COIN and Breakfast, RLT reduces total tokens by 71% and 79% respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard tokenization with RLT | ViT-B · Kinetics-400 | accuracy: 80.1 -> 80.1 | No clear effect |
| replace standard tokenization with RLT | ViT-B · Kinetics-400 | FT time (8 GPU): 14.4h -> 10.2h | Helped |
| replace standard tokenization with RLT | ViT-B · Something-Something-v2 | accuracy: 70.3 -> 70.2 | No clear effect |
| replace standard tokenization with RLT | ViT-L · Kinetics-400 | accuracy: 84.8 -> 84.7 | No clear effect |
| replace standard tokenization with RLT | ViT-L · Something-Something-v2 | accuracy: 74.3 -> 74.4 | No clear effect |
| replace standard tokenization with RLT | ViT-B · Kinetics-400 inference | accuracy: 80.5 -> 80.6 | No clear effect |
| replace standard tokenization with RLT | ViT-B · Kinetics-400 inference | clips/s: 31.4 -> 52.6 | Helped |
| replace standard tokenization with RLT | ViT-L · Kinetics-400 inference | accuracy: 84.8 -> 84.6 | No clear effect |
| use RLT instead of standard tokenization | ViT-L · Kinetics-400, 30 FPS | FT time: 110h -> 52.3h | Helped |
| increase the difference threshold above 0.1 (ablation) | RLT · action recognition | accuracy | Hurt |
| lower the difference threshold below 0.1 (ablation) | RLT · action recognition | speedup | Hurt |
| add length encoding (ablation) | RLT · action recognition | accuracy | No clear effect |
Surprising findings
- Length encoding has minimal effect when RLT is used alone, but helps notably when combined with random masking.
- Random masking at the matched token-reduction ratio performs surprisingly well, suggesting most video tokens are redundant.
Limitations
- Relies on a pixel-difference heuristic that may keep tokens unused by the transformer.
- Cannot be used for dense prediction tasks requiring one output token per input token, such as tracking or video generation.
- Handles camera motion poorly; constant camera motion removes few tokens and gives no speedup.
Experimental setup
Datasets: Kinetics-400 Something-Something-v2 Breakfast COIN EPIC-Kitchens-100
Benchmarks: Kinetics-400 Something-Something-v2 EPIC-Kitchens-100 COIN Breakfast
Metrics: top-1 accuracy wall-clock fine-tuning time throughput (clips/s) GFLOPS token count reduction
Baselines: Token Merging (ToMe) random masking STA ViT baseline
Builds on: VideoMAE Token Merging Vision Transformer NaViT
Closest papers by meaning
- vid-TLDR: Training Free Token Merging for Light-weight Video Transformer (CVPR 2024 · similarity 0.88)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.88)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.87)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.87)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.