Research map / Video / Video-language understanding
DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free two-stage token compression for video LLMs: temporal token merging at prefill plus dynamic KV cache pruning during decoding, yielding ~1.5x speedup and 1.4x memory reduction on LLaVA-OneVision.
Problem
Video LLM inference is expensive because thousands of visual tokens are generated from video frames; one-shot pruning removes tokens that later decoding steps need, since attention to visual tokens shifts across decoding iterations.
Approach
Combine a plug-and-play temporal token merging module (cosine similarity between sampled frame groups) at prefill with a decoding-time dynamic KV cache pruning scheme that keeps a DP cache of pruned tokens so they can be recalled when attention shifts back.
Key results
- On LLaVA-OV-7B, DyCoke (K=0.5) reaches 57.88% MVBench accuracy vs 57.58% full tokens with 1.35x per-example speedup and 21G vs 27G GPU memory (16 frames).
- With 32 frames on LLaVA-OV-7B, DyCoke (K=0.7) achieves 1.54x per-example speedup and 24G vs 34G GPU memory, with accuracy 59.56% vs 58.81% for full tokens.
- On VideoMME, DyCoke (K=0.5) on LLaVA-OV-72B scores 60.2 (w/o subs) vs 66.9 for full tokens at a 60% FLOPs ratio, outperforming FastV and PruMerge at comparable cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DyCoke token compression | LLaVA-OV-7B · MVBench (32 frames) | latency per example: 2.30s (1.00×) -> 1.49s (1.54×) | Helped |
| Apply DyCoke token compression | LLaVA-OV-7B · MVBench (32 frames) | accuracy: 58.81 (±0.14) -> 59.56 (±0.19) | Helped |
| Add dynamic pruning during decoding (ablation) | DyCoke on LLaVA-OV-7B · ActivityNet-QA | accuracy: 51.06 -> 51.80 | Helped |
| Add dynamic pruning during decoding (ablation) | DyCoke on LLaVA-OV-0.5B · ActivityNet-QA | accuracy: 46.44 -> 47.70 | Helped |
| Replace dynamic pruning with random token pruning (ablation) | DyCoke on LLaVA-OV-7B · ActivityNet-QA | accuracy: 51.80 -> 50.90 | Hurt |
| Replace temporal similarity-based selection with random pruning (ablation) | DyCoke on LLaVA-OV-0.5B · ActivityNet-QA | accuracy: 47.70 -> 41.68 | Hurt |
| Change attention evaluation layer from L=3 to L=10 (ablation) | DyCoke on LLaVA-OV-7B · ActivityNet-QA | accuracy: 51.80 -> 51.81 | No clear effect |
| Increase second-stage pruning rate from P=0.7 to P=0.9 (ablation) | DyCoke on LLaVA-OV-7B · ActivityNet-QA | accuracy: 51.80 -> 51.48 | Hurt |
| Increase first-stage pruning to K=0.9 and use L=0, P=0.9 (ablation) | DyCoke on LLaVA-OV-7B · ActivityNet-QA | accuracy: 51.80 -> 40.21 | Hurt |
| Increase first-stage pruning to K=0.9 and use L=0, P=0.9 (ablation) | DyCoke on LLaVA-OV-0.5B · VideoDetailCaption | score: 2.56 -> 1.75 | Hurt |
| Use DyCoke to process 32 rather than 16 frames | LLaVA-OV-7B · VideoMME | average accuracy: 56.2 -> 58.3 | Helped |
Surprising findings
- Removing dynamic pruning (one-shot pruning) causes the largest drop on VideoDetailCaption, showing that decoding-time re-selection matters for long generation.
- Pruned tokens in several benchmarks slightly improve over full-token accuracy (e.g., PerceptionTest and VideoMME on LLaVA-OV-7B), suggesting redundancy removal can help.
Limitations
- Gains are reported mainly against two training-free baselines (FastV, PruMerge) on a single model family (LLaVA-OneVision).
- Evaluation of open-ended tasks relies on GPT-4o-mini as judge.
Experimental setup
Datasets: ActivityNet-QA PerceptionTest VideoMME NeXT-QA VideoDetailCaption MVBench
Benchmarks: ActivityNet-QA PerceptionTest VideoMME NeXT-QA VideoDetailCaption MVBench
Metrics: accuracy score (0-5) FLOPs latency GPU memory
Baselines: FastV LLaVA-PruMerge Full tokens
Builds on: FastV LLaVA-PruMerge ToMe LLaVA-OneVision
Closest papers by meaning
- Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior (NeurIPS 2025 · similarity 0.94)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.91)
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.91)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
- PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.