Research map / Video / Video-language understanding
HoliTom: Holistic Token Merging for Fast Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HoliTom, a training-free holistic token merging framework combining global redundancy-aware temporal segmentation plus spatial merging before the LLM with similarity-based token merging inside the LLM, to speed up video LLMs.
Problem
Video LLMs produce tens of thousands of visual tokens, making prefilling and decoding expensive; existing inner-LLM and outer-LLM pruning methods are either costly in shallow layers or only exploit limited temporal windows, and their combination is unexplored.
Approach
Combines outer-LLM global dynamic-programming temporal segmentation and redundancy-aware spatio-temporal merging (attention-based selection for non-redundant tokens, DPC-KNN clustering for temporally redundant tokens) with an inner-LLM merging that averages low-attention tokens into retained ones at a chosen layer instead of discarding them.
Key results
- At 10% retained tokens before the LLM on LLaVA-OneVision-7B, HoliTom reaches 6.9% of FLOPs while keeping 99.1% of average performance (57.9 vs 58.4 vanilla).
- At 25% retention, HoliTom (w/o inner merging) achieves 100.5% average performance at 21.3% FLOPs, outperforming VisionZip (99.7%) and PruneVid (98.6%) at the same budget.
- Reports 2.28x TTFT reduction and 1.32x decoding throughput improvement over the vanilla model on LLaVA-OneVision-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HoliTom compression at 10% before-LLM retained ratio | LLaVA-OneVision-7B · Four video benchmarks (average) | Avg. score: 100 -> 99.1 | No clear effect |
| Add inner-LLM merging to outer-LLM merging (ablation) | HoliTom · Four video benchmarks (average), 15% retained ratio | Prefilling FLOPs: 5.2 -> 4.3 | Helped |
| Add inner-LLM merging to outer-LLM merging (ablation) | HoliTom · Four video benchmarks (average), 15% retained ratio | Avg. score: 58.5 -> 58.2 | Hurt |
| Use temporal merging only rather than the full spatial-temporal method (ablation) | HoliTom · Four video benchmarks (average) | FLOPs ratio: 10.5% -> 75.7% | Hurt |
| Add temporal merging to spatial-only merging (ablation) | HoliTom · LongVideoBench | score: 54.2 -> 57.0 | Helped |
| Use attention-based selection and DPC-KNN together rather than either strategy alone (ablation) | HoliTom · LongVideoBench | score: 55.9 -> 56.2 | Helped |
| Replace attention-based merging with DPC-KNN only (ablation) | HoliTom · LongVideoBench | score: 55.9 -> 53.9 | Hurt |
| Set temporal similarity threshold to around 0.8 (ablation) | HoliTom · Video benchmarks at 15% retained ratio | performance | Helped |
| Use temporal segmentation that globally maximizes redundant-token pruning (ablation) | HoliTom · Video understanding benchmarks | performance | Helped |
Surprising findings
- Token compression often improves over the uncompressed model; e.g., HoliTom at 25% retention scores above the vanilla LLaVA-OV-7B average.
- Temporal-only merging reduces FLOPs to 75.7% while slightly increasing average performance (100.7%).
Limitations
- Designed for fixed-length video clips; does not natively support online streaming video.
- Does not optimize vision tower latency, which remains a cost.
Experimental setup
Datasets: MVBench EgoSchema LongVideoBench VideoMME
Benchmarks: MVBench EgoSchema LongVideoBench VideoMME
Metrics: accuracy prefilling FLOPs FLOPs ratio time-to-first-token decoding throughput
Baselines: FastV PDrop VisionZip DyCoke PruneVid FastVID
Builds on: FastV VisionZip DyCoke DPC-KNN
Closest papers by meaning
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.92)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.91)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs (NeurIPS 2026 · similarity 0.89)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.89)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.