Research map / Video / Video-language understanding
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongVU, a spatiotemporal adaptive token compression method (DINOv2 similarity-based frame reduction, text-query-guided selective token reduction, and spatial token compression) that lets an MLLM process 1fps long videos within an 8k context.
Problem
Long video understanding in MLLMs is constrained by LLM context length; uniform sampling loses key frames and aggressive resampling loses visual detail, so hour-long videos cannot be fully represented within an 8k budget.
Approach
Combine three compression steps: remove temporally redundant frames using DINOv2 feature similarity, keep full spatial tokens only for frames relevant to the text query (cross-modal attention) while pooling others, and prune spatial tokens that are redundant relative to the first frame in each temporal window (STC), fusing SigLIP and DINOv2 features via SVA.
Key results
- LongVU (7B, 8k context, 1fps) reaches 60.6 overall and 59.5 long-subset on VideoMME, versus 58.2 and 46.7 for LLaVA-OneVision (7B, 32 frames).
- LongVU reaches 67.6 on EgoSchema and 66.9 on MVBench, beating GPT-4o on MVBench (64.6).
- With Llama3.2-3B, LongVU reaches 51.5 overall and 47.2 long on VideoMME, outperforming Phi-3.5-vision-instruct (50.8/43.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LongVU instead of LLaVA-OneVision | 7B video MLLM · VideoMME Long | accuracy: 46.7 -> 59.5 | Helped |
| Use DINO features instead of SigLIP features (ablation) | 8k uniform 64-token video setting · VideoMME | accuracy: 58.63 -> 59.90 | Helped |
| Use DINO-based temporal frame reduction instead of uniform sampling (ablation) | 16k, 144-token video setting · VideoMME | accuracy: 60.01 -> 61.25 | Helped |
| Add query-guided frame selection after DINO temporal reduction (ablation) | LongVU · EgoSchema | accuracy: 66.20 -> 67.30 | Helped |
| Add query-guided frame selection after DINO temporal reduction (ablation) | LongVU · VideoMME | accuracy: 59.90 -> 60.08 | Helped |
| Add query-guided frame selection after DINO temporal reduction (ablation) | LongVU · MLVU | accuracy: 62.54 -> 65.05 | Helped |
| Add spatial token compression (STC) after DINO + Query (ablation) | LongVU · EgoSchema | accuracy: 67.30 -> 67.62 | Helped |
| Add spatial token compression (STC) after DINO + Query (ablation) | LongVU · VideoMME | accuracy: 60.08 -> 60.56 | Helped |
| Add spatial token compression (STC) after DINO + Query (ablation) | LongVU · MLVU | accuracy: 65.05 -> 65.44 | Helped |
| Add frame-level positional encoding (FPE) (ablation) | LongVU with DINO + Query + STC · MLVU | accuracy: 65.44 -> 64.56 | Hurt |
| Use the first frame rather than the middle frame as the STC anchor (ablation) | LongVU · VideoMME overall | accuracy: 60.7 -> 60.9 | Helped |
| Use the first frame rather than the high-change frame as the STC anchor (ablation) | LongVU · VideoMME overall | accuracy: 60.4 -> 60.9 | Helped |
Surprising findings
- Video-only SFT degrades image understanding noticeably (e.g., SQA-IMG 95.44 to 83.94, MMVP 51.33 to 32.00).
- Adding frame-level positional encoding did not consistently help and was left out of the default setting.
Limitations
- Video-only SFT causes a drop in image understanding performance.
- Training uses a relatively small video dataset compared with LLaVA-OneVision's unreleased OneVision-1.6M data.
- Evaluation focuses on a few video benchmarks and the gains from STC are modest over the query-guided variant.
Experimental setup
Datasets: LLaVA-OneVision single-image data VideoChat2-IT subset TextVR YouCook2 Kinetics-710 NExTQA CLEVRER EgoQA TGIF WebVidQA ShareGPT4Video MovieChat
Benchmarks: EgoSchema MVBench VideoMME MLVU Needle-in-a-Haystack (video)
Metrics: accuracy
Baselines: GPT-4o GPT-4V LLaVA-OneVision VideoChat2 LongVA LLaVA-NeXT-Video VideoLLaMA2 Chat-UniVi Phi-3.5-vision-instruct InternVL2
Builds on: LLaVA-OneVision Cambrian-1 SigLIP DINOv2 Spatial Vision Aggregator
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.92)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.90)
- Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.89)
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.