Research map / Video / Video-language understanding
Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyToK, a training-free method that uses attention from a lightweight assistant VLLM (deep layers) to estimate per-frame importance and dynamically allocate visual token budgets before plugging into existing pruning methods (VisionZip, FastV, DyCoke).
Problem
Long-video VLLMs suffer quadratic cost from long visual token sequences; existing keyframe selection adds pre-encoding overhead and binary frame selection discards useful content, while fixed per-frame compression ignores temporal importance.
Approach
Use query-conditioned cross-modal attention from deep layers of a small same-family assistant VLLM as a keyframe prior, and dynamically allocate per-frame token budgets proportional to these scores, then apply any existing token pruning method per frame.
Key results
- On LLaVA-OneVision-7B with 32 frames at 50% retention, VisionZip+DyToK reaches 99.4% of vanilla average vs 97.4% for VisionZip alone; gain of +4.2 points over VisionZip† at 50% retention.
- At 10% retention (448 tokens) with 32 frames, DyToK improves VisionZip† from 83.4% to 90.2% of vanilla average accuracy (+18.9 points for 90% compression as reported).
- On 64-frame LLaVA-OneVision, DyToK (7B guidance) reaches 93.7% of vanilla at 90% compression vs 81.8% for VisionZip†, up to 24.0 points gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DyToK to VisionZip† | VisionZip† on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 50% token reduction | average score: 53.4 -> 53.9; (+4.2) | Helped |
| Add DyToK to VisionZip† | VisionZip† on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 90% token reduction | average score: 45.1 -> 48.8; (+18.9) | Helped |
| Add DyToK to FastV | FastV on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 85% token reduction | average score: 46.9 -> 50.2; (+6.1) | Helped |
| Add DyToK to FastV | FastV on Qwen2.5-VL-7B · VideoMME, LongVideoBench, and MLVU; 50% token retention | average score: 48.6 -> 54.3; (+10.5) | Helped |
| Use DyToK instead of VisionZip alone | VisionZip on VideoChatGPT · VideoChatGPT; 10% retention | average score: 1.53 -> 2.37 | Helped |
| Use deeper layers for keyframe-prior attention (ablation) | DyToK on LLaVA-OneVision · VideoMME; 20% retention | overall accuracy: 54.0 -> 57.5 | Helped |
| Use LLaVA-OneVision-0.5B instead of LLaVA-OneVision-7B as assistant (ablation) | DyToK on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 25% token reduction | average score: 54.1 -> 53.8; (-0.6) | Hurt |
| Use LLaVA-OneVision-0.5B instead of LLaVA-OneVision-7B as assistant (ablation) | DyToK on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 50% token reduction | average score: 53.7 -> 53.9; (+0.3) | No clear effect |
| Use LLaVA-OneVision-0.5B instead of LLaVA-OneVision-7B as assistant (ablation) | DyToK on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 75% token reduction | average score: 54.1 -> 53.3; (-1.5) | Hurt |
| Use LLaVA-OneVision-0.5B instead of LLaVA-OneVision-7B as assistant (ablation) | DyToK on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 80% token reduction | average score: 53.5 -> 53.0; (-0.9) | Hurt |
| Use LLaVA-OneVision-0.5B instead of LLaVA-OneVision-7B as assistant (ablation) | DyToK on LLaVA-OneVision · VideoMME, LongVideoBench, and MLVU; 90% token reduction | average score: 49.6 -> 48.8; (-1.5) | Hurt |
| Replace cross-modal attention frame weights with CLIP similarity (ablation) | DyToK on LLaVA-OneVision · VideoMME; 25% retention | overall accuracy: 58.3 -> 58.1 | Hurt |
Surprising findings
- Using a 0.5B same-family assistant model to provide keyframe priors performs comparably to the 7B model, despite being 14x smaller.
- Deeper attention layers give better keyframe priors than shallow layers, and using the deepest layer for the first layer nearly doubles inference cost.
Limitations
- Still requires running an additional lightweight assistant model, which adds overhead and is not eliminated.
- Gains are small or negative for some configurations (e.g., Qwen2.5-VL with VisionZip at 75% and 15% retention shows slight drops).
Experimental setup
Datasets: VideoMME LongVideoBench MLVU VideoChatGPT
Benchmarks: VideoMME LongVideoBench MLVU VideoChatGPT captioning
Metrics: accuracy average score relative accuracy percentage GPT-based quality score
Baselines: FastV VisionZip DyCoke CLIP-based keyframe selection (AKS-style)
Builds on: FastV VisionZip DyCoke AKS
Closest papers by meaning
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.94)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.93)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.91)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.89)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.