Research map / Video / Video-language understanding
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video-XL, a 7B video MLLM that compresses long videos into KV caches of learned Visual Summarization Tokens (VST), trained with curriculum over compression ratios, composite image/multi-image/synthetic data, and dynamic interval partitioning.
Problem
Long video understanding in MLLMs is limited by context length and compute cost, and existing visual token reduction methods lose fine-grained visual detail.
Approach
Instead of reducing visual tokens before the LLM, interleave learned Visual Summarization Tokens into each video interval so the LLM's own KV cache summarizes the interval; the VST KVs replace the original visual KVs for subsequent intervals. Interval sizes follow semantic depth scores (dynamic compression), and training uses curriculum over compression ratios plus composite data.
Key results
- On MLVU Dev, Video-XL (7B) scores 64.9 M-avg, above GPT-4o's 64.6 and the best open-source baselines.
- On Video-MME, Video-XL reaches 55.5% without subtitles and 61.0% with subtitles.
- Video-XL handles 2048 frames on a single A100-80GB and keeps about 95% Needle-in-a-Haystack accuracy at longer inputs; 100% within 128 frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Video-XL instead of LongVA | 7B video MLLMs · MLVU Dev | M-avg: 56.3 -> 64.9 | Helped |
| Use Video-XL instead of Video-CCAM | 7B video MLLMs · MLVU Test | M-avg: 42.9 -> 45.5 | Helped |
| Use Video-XL instead of LongVA | 7B video MLLMs · VideoMME without subtitles | accuracy: 52.6 -> 55.5 | Helped |
| Use Video-XL instead of Long-LLaVA | open-source video MLLMs · VNBench | accuracy: 52.1 -> 61.6 | Helped |
| Use Video-XL instead of GPT-4o | video MLLMs · VideoVista | score: 78.3 -> 70.6 | Hurt |
| Use Video-XL instead of Long-LLaVA | video MLLMs · LongVideoBench | score: 52.1 -> 50.7 | Hurt |
| Use Video-XL instead of LongVA | video MLLMs · VideoChatGPT Benchmark | score: 3.26 -> 3.17 | Hurt |
| Replace VST compression with pooling at 16× compression (ablation) | Video-XL compression · MLVU-test | score: 41.4 -> 33.7 | Hurt |
| Replace VST compression with Q-Former at 16× compression (ablation) | Video-XL compression · MLVU-test | score: 41.4 -> 35.1 | Hurt |
| Replace VST compression with C-Abstractor at 16× compression (ablation) | Video-XL compression · MLVU-test | score: 41.4 -> 37.1 | Hurt |
| Add dynamic compression during training and testing (ablation) | Video-XL · MLVU | score: 39.8 -> 41.6 | Helped |
| Enable dynamic compression only at test time (ablation) | Video-XL · MLVU | score: 39.8 -> 39.6 | No clear effect |
Surprising findings
- Dynamic compression helps only when enabled during training; enabling it only at test time gave no improvement.
- VICO with only 5k samples outperformed NExT-QA (32k) and CinePile (10k) training sets on Video-MME and MLVU.
Limitations
- Compression quality degrades for unseen compression ratios beyond training range unless tested without fine-tuning (32x tested without fine-tuning)
- Long-video instruction data is scarce, relying on synthetic and image data substitutes
- Multi-detail action order gains from image data are marginal
Experimental setup
Datasets: Laion-2M Bunny ShareGPT-4o MMDU NExT-QA CinePile VCG VICO
Benchmarks: MLVU Video-MME VNBench LongVideoBench VideoVista VideoChatGPT Bench MVBench MME MMBench MLVU-test
Metrics: accuracy GPT score retention rate
Baselines: LongVA LLaMA-VID VideoChat2 LLaVA-NeXT-Video Long-LLaVA Video-CCAM GPT-4o Gemini-1.5-Pro
Builds on: LLaVA VideoLLaMB LongVA LLaMA-VID
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.92)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.90)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.