Research map / Video / Video segmentation and tracking
vid-TLDR: Training Free Token Merging for Light-weight Video Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes vid-TLDR, a training-free token merging method for video Transformers that uses attention sharpness to detect salient tokens and merges background tokens early, reducing FLOPs while keeping retrieval and QA performance.
Problem
Video Transformers incur heavy quadratic attention cost from the huge number of spatio-temporal tokens, and background tokens degrade generalization; prior training-free image token reduction methods are unreliable in early layers and biased temporally in video.
Approach
Estimate token saliency from the sharpness (negative entropy) of attention rows in the first layer, then perform saliency-aware token merging that zeroes background masses and rescales foreground masses so background tokens barely influence merged features.
Key results
- On UMT-B, vid-TLDR improves MSRVTT text-to-video/video-to-text average R@1 from 50.0 to 50.8 while reducing GFLOPs from 303.3 to 178.0.
- Without training, vid-TLDR on UMT-B-based video QA keeps MSRVTT-QA at 44.8 vs 44.9 base with about 38% fewer FLOPs.
- On VideoMAE SSV2, vid-TLDR reduces GFLOPs from 180.5 to 56.6 with accuracy 69.6 vs 70.8 base, outperforming ToMe (58.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply vid-TLDR token merging | UMT-B · MSRVTT | R@1: 50.0 -> 50.8 | Helped |
| apply vid-TLDR token merging | UMT-B · MSVD | R@1: 62.1 -> 62.7 | Helped |
| apply vid-TLDR token merging | UMT-B · DiDeMo | R@1: 62.1 -> 62.4 | Helped |
| replace baseline token reduction with vid-TLDR | UMT-B · MSRVTT | R@1: 47.0 -> 50.8 | Helped |
| apply vid-TLDR | UMT-B · MSRVTT-QA | accuracy: 44.9 -> 44.8 | No clear effect |
| apply vid-TLDR | UMT-B · MSVD-QA | accuracy: 49.5 -> 49.4 | No clear effect |
| apply vid-TLDR | VideoMAE · UCF101 | Acc: 91.3 -> 90.1 | Hurt |
| apply vid-TLDR | VideoMAE · SSV2 | Acc: 70.8 -> 69.6 | Hurt |
| use attentiveness scores for token pruning instead of saliency-aware merging (ablation) | UMT-B · MSRVTT | Mean retrieval score: 50.0 -> 49.0 | Hurt |
| use attention rollout scores for token pruning instead of saliency-aware merging (ablation) | UMT-B · MSRVTT | Mean retrieval score: 50.0 -> 49.3 | Hurt |
| use attention-sharpness saliency scores for token pruning (ablation) | UMT-B · MSRVTT | Mean retrieval score: 50.0 -> 50.3 | Helped |
| add saliency-aware token merging to saliency-score pruning (ablation) | UMT-B · MSRVTT | Mean retrieval score: 50.3 -> 50.8 | Helped |
Surprising findings
- Pruning by attention-based saliency in the first layer gives the best results, showing early token reduction is more effective than later layers.
- Attentiveness and attention rollout show a temporal bias where later frames receive substantially lower scores, which the sharpness-based score mitigates.
Limitations
- Performance is usually slightly below the base model on several datasets (e.g., UMT-L on several benchmarks), so gains are not uniform.
- Relies on the attention map of the early layer for saliency, and the method's behavior on non-transformer or non-attention architectures is untested.
Experimental setup
Datasets: MSRVTT MSVD ActivityNet DiDeMo LSMDC SSV2 MSRVTT-QA MSVD-QA UCF101
Benchmarks: MSRVTT MSVD ActivityNet DiDeMo LSMDC SSV2-Label SSV2-Template MSRVTT-QA MSVD-QA UCF101
Metrics: R@1 R@5 R@10 GFLOPs accuracy harmonic mean
Baselines: UMT ToMe VideoMAE ViFi-CLIP attentiveness attention rollout
Builds on: ToMe UMT attention rollout
Closest papers by meaning
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.91)
- Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (NeurIPS 2024 · similarity 0.88)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.88)
- VidToMe: Video Token Merging for Zero-Shot Video Editing (CVPR 2024 · similarity 0.87)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.86)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.