Research map / Video / Video-language understanding
Video Token Merging for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores video token merging variants for long-form video classification and proposes learnable video token merging (VTM), which merges tokens by learned saliency scores, cutting memory by 84% and raising throughput ~6.89x.
Problem
Transformers on long-form video are prohibitively expensive because self-attention cost grows quadratically with token count; existing token merging relies only on feature similarity and uniform partitioning, which can merge salient tokens away.
Approach
Extends token merging to video with saliency-aware partitioning: explores naive, center/boundary-concentrated, and motion-based (codec motion vectors) merging, then learns a per-token saliency score via an auxiliary saliency-guided attention path used only during training, so that salient tokens are kept unmerged at inference.
Key results
- On LVU, learnable VTM achieves best scores on 7 of 9 tasks, e.g. Relationship 64.28, Director 70.09, Year 48.55, with 1.60GB GPU memory and 44.94 throughput.
- On Breakfast, learnable VTM reaches 91.26% accuracy, beating S5+LSMCL (90.70%) and D-sprv (89.90%) with Kinetics-600 pretraining.
- Compared with baseline at gamma=6, memory is reduced by 84% (10GB to 1.6GB) and throughput is raised about 6.89x (6.52 to 44.94).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the baseline with learnable video token merging | ViT-L long-video classifier · LVU | Scene accuracy: 69.76 -> 75.58 | Helped |
| Replace the baseline with learnable video token merging | ViT-L long-video classifier · LVU | GPU memory: 10GB -> 1.6GB | Helped |
| Use learnable video token merging instead of S5+LSMCL | Long-video classifier · Breakfast | Accuracy: 90.70 -> 91.26 | Helped |
| Use learnable video token merging instead of S5+LSMCL | Long-video classifier · COIN | Accuracy: 90.81 -> 88.55 | Hurt |
| Concentrate target tokens in the center rather than use uniform partitioning (ablation) | Video token merging · LVU | Scene accuracy: 72.09 -> 74.41 | Helped |
| Concentrate target tokens at boundaries rather than use uniform partitioning (ablation) | Video token merging · LVU | Scene accuracy: 72.09 -> 69.76 | No clear effect |
| Use motion-based target-token selection rather than uniform partitioning (ablation) | Video token merging · LVU | Scene accuracy: 72.09 -> 74.41 | Helped |
| Set γ=6 rather than γ=2 (ablation) | Learnable VTM · LVU | Scene accuracy: 72.09 -> 75.58 | Helped |
| Set γ=10 rather than γ=6 (ablation) | Learnable VTM · LVU | Throughput: 44.94 -> 48.89 | Helped |
| Use temporal grouping (4, 20, 60) instead of (6, 30, 60) (ablation) | Learnable VTM · LVU | Scene accuracy: 75.58 -> 73.25 | Hurt |
| Use temporal grouping (10, 30, 60) instead of (6, 30, 60) (ablation) | Learnable VTM · LVU | Throughput: 44.94 -> 33.62 | Hurt |
| Use size-weighted average pooling instead of average pooling (ablation) | Learnable VTM · LVU | Scene accuracy: 75.58 -> 72.09 | Hurt |
Surprising findings
- Weighted average pooling by token size reduced learnable VTM performance on LVU (Scene 75.58 to 72.09), so plain average pooling was kept.
- On COIN, learnable VTM (88.55%) is only comparable to S5 (90.42%) and S5+LSMCL (90.81%), and the paper notes that missing YouTube videos make the comparison imperfect.
Limitations
- Comparison on COIN is not fully fair because many videos are unavailable, so results are reported only for reference.
- Learnable VTM underperforms on some LVU tasks (e.g., Genre, User View) compared with S5 variants.
- Experiments are limited to classification/regression on three datasets with a three-block baseline.
Experimental setup
Datasets: LVU Breakfast COIN ImageNet-21K Kinetics-600
Benchmarks: LVU Breakfast COIN
Metrics: top-1 classification accuracy mean-squared error throughput GPU memory
Baselines: Object T4mer VideoBERT Performer Orthoformer LST ViS4mer S5 S5+LSMCL D-sprv VideoGraph Timeception GHRM
Builds on: Token Merging (ToMe) ViS4mer S5 Object Transformer
Closest papers by meaning
- vid-TLDR: Training Free Token Merging for Light-weight Video Transformer (CVPR 2024 · similarity 0.91)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.89)
- FlexSelect: Flexible Token Selection for Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Efficient Time Series Processing for Transformers and State-Space Models through Token Merging (ICML 2025 · similarity 0.88)
- VidToMe: Video Token Merging for Zero-Shot Video Editing (CVPR 2024 · similarity 0.88)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.