Research map / Video / Video-language understanding
One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses each sampled video frame to one token inside the LLM layers via progressive, learned suffix-preserving compression (LP-Comp), plus segment-local question-conditioned frame selection (QC-Comp), fine-tuned from VideoChat-Flash-2B.
Problem
Long video understanding in VLMs is limited by LLM context length, since each frame yields tens to hundreds of visual tokens, forcing sparse frame sampling and loss of temporal information.
Approach
Supervise LLM layers to progressively compress visual tokens (cosine schedule, suffix-preserving) down to one token per frame during a short supervised compression tuning stage, and select relevant frames using question-to-video attention computed locally within 64-frame segments to mitigate LLM position bias.
Key results
- XComp built on VideoChat-Flash-2B raises LVBench accuracy from 42.9 to 46.2 and LongVideoBench from 58.3 to 59.7 using 2.5% of SFT data.
- Learnable progressive compression beats training-free heuristic compression on LVBench at 1024 frames (44.3 vs 39.7 for progressive; heuristic drops to 41.1 vs 41.8 baseline).
- Segment-local attention for QC-Comp beats global attention (LongVideoBench 59.7 vs 59.5; VideoMME long 45.7 vs 44.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune VideoChat-Flash-2B with XComp compression | VideoChat-Flash-2B · LongVideoBench | accuracy: 58.3 -> 59.7 | Helped |
| fine-tune VideoChat-Flash-2B with XComp compression | VideoChat-Flash-2B · MLVU | accuracy: 65.7 -> 66.7 | Helped |
| fine-tune VideoChat-Flash-2B with XComp compression | VideoChat-Flash-2B · VideoMME (Long) | accuracy: 44.9 -> 45.6 | Helped |
| fine-tune VideoChat-Flash-2B with XComp compression | VideoChat-Flash-2B · LVBench | accuracy: 42.9 -> 46.2 | Helped |
| apply heuristic token compression without learning (ablation) | VideoChat-Flash · LVBench (1024 frames) | accuracy: 41.8 -> 41.1 | Hurt |
| use learnable rather than training-free compression (ablation) | Step-wise token compression · LVBench (1024 frames) | accuracy: 38.4 -> 42.3 | Helped |
| use learnable rather than training-free compression (ablation) | Progressive token compression · LVBench (1024 frames) | accuracy: 39.7 -> 44.3 | Helped |
| use progressive rather than step-wise token compression (ablation) | Learnable token compression · LVBench (1024 frames) | accuracy: 42.3 -> 44.3 | Helped |
| preserve suffix tokens rather than drop tokens uniformly (ablation) | XComp LP-Comp · LVBench | accuracy: 43.6 -> 44.3 | Helped |
| preserve suffix tokens rather than drop tokens uniformly (ablation) | XComp LP-Comp · MLVU | accuracy: 64.1 -> 65.2 | Helped |
| use segment-local rather than global attention for frame scoring (ablation) | XComp QC-Comp · LongVideoBench | accuracy: 59.5 -> 59.7 | Helped |
| use segment-local rather than global attention for frame scoring (ablation) | XComp QC-Comp · MLVU | accuracy: 66.3 -> 66.7 | Helped |
Surprising findings
- Heuristic token compression from VideoChat-Flash performs worse than the uncompressed baseline at this extreme compression ratio (41.1 vs 41.8 on LVBench).
- LP-Comp alone yields only a small gain on LongVideoBench (58.3 to 58.8), with most of the gain coming from combining with QC-Comp.
Limitations
- Only experiments with VideoChat-Flash-2B and LLaVA-Next-Video at small scale; not tested on larger VLMs.
- No error bars or multiple seeds reported due to compute budget.
- Frame selection is not jointly trained with the LLM; future work proposes joint optimization.
Experimental setup
Datasets: VideoChat-Flash SFT data (2.5% subset) LLaVA-Video-178k (2.5% subset for LLaVA-Next-Video)
Benchmarks: LongVideoBench MLVU LVBench VideoMME (long, w/o subtitles)
Metrics: accuracy
Baselines: VideoChat-Flash-2B InternVL3-2B VideoChat-Flash-7B GPT-4o Gemini-1.5-Pro LLaVA-Next-Video
Builds on: VideoChat-Flash LLaVA-NeXT-Video LLaVA
Closest papers by meaning
- Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior (NeurIPS 2025 · similarity 0.93)
- Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025 · similarity 0.92)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.92)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.91)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.