Research map / Video / Video-language understanding
Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViLaMP, a hierarchical 7B video-language model that uses differential keyframe selection and differential feature merging to process up to 10K frames on a single A100 GPU.
Problem
Long-form video understanding in VLMs is bottlenecked by the huge number of visual tokens; existing token pruning and feature merging either lose temporal dependencies or dilute semantic information.
Approach
A differential saliency score D = relevance to query minus temporal redundancy, applied at frame level (greedy keyframe selection with a similarity threshold) and patch level (softmax-weighted pooling of non-keyframe patches into one token), so keyframes keep full tokens and non-keyframes are compressed to a single token.
Key results
- ViLaMP reaches 46.3 on LVBench, 74.2 on EgoSchema, 60.2 on LongVideoBench and 67.7/71.6 on Video-MME overall (wo/w subs) among similar-sized open models.
- On VideoNIAH at 10K frames, ViLaMP averages 58.15% versus 47.25% for VideoChat-Flash, and processes up to 10K frames on one A100.
- Ablation on LongVideoBench/MLVU/Video-MME: DKS and DFM together raise LLaVA-OneVision from 56.5/64.7/58.2 to 60.5/72.6/65.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add differential keyframe selection (DKS) (ablation) | LLaVA-OneVision · MLVU | accuracy: 64.7 -> 67.3 | Helped |
| Add differential keyframe selection (DKS) (ablation) | LLaVA-OneVision · Video-MME | accuracy: 58.2 -> 61.8 | Helped |
| Replace DKS with query-guided sampling (ablation) | LLaVA-OneVision · Video-MME | accuracy: 61.8 -> 55.6 | Hurt |
| Replace DKS with uniform sampling (ablation) | LLaVA-OneVision · Video-MME | accuracy: 61.8 -> 60.2 | Hurt |
| Add differential feature merging (DFM) (ablation) | LLaVA-OneVision · LongVideoBench Validation | accuracy: 56.5 -> 60.5 | Helped |
| Add differential feature merging (DFM) (ablation) | LLaVA-OneVision · MLVU | accuracy: 64.7 -> 72.6 | Helped |
| Replace DFM with Q-former (ablation) | LLaVA-OneVision · LongVideoBench Validation | accuracy: 60.5 -> 53.9 | Hurt |
| Replace DFM with mean-pooling (ablation) | LLaVA-OneVision · LongVideoBench Validation | accuracy: 60.5 -> 56.3 | Hurt |
| Increase the keyframe count; performance plateaus around 16 for short videos and requires 32 for longer videos (ablation) | ViLaMP · Video-MME-Short, LongVideoBench, Video-MME-Medium/Long, MLVU | accuracy | Helped |
| Set the keyframe similarity threshold τ to 0.85 rather than 0.35 or 0.50 (ablation) | ViLaMP · LongVideoBench, MLVU, Video-MME | accuracy: 56.8 -> 60.5; 65.3 -> 72.6; 60.2 -> 65.1 | Helped |
| Set the keyframe similarity threshold τ to 1.00 instead of 0.85 (ablation) | ViLaMP · LongVideoBench, MLVU, Video-MME | accuracy: 60.5 -> 50.9; 72.6 -> 62.3; 65.1 -> 58.8 | Hurt |
| Set feature-merging temperature α to 10^-2 rather than 10^-3 (ablation) | ViLaMP · LongVideoBench, MLVU, Video-MME | accuracy: 54.4 -> 60.5; 65.6 -> 72.6; 48.2 -> 65.1 | Helped |
Surprising findings
- Query-guided top-k frame sampling underperforms uniform sampling on several benchmarks (e.g., 55.6 vs 60.2 on Video-MME), suggesting relevance alone yields redundant frames.
Limitations
- Training cost is large (about two weeks on 32 A100 GPUs for one epoch).
- Evaluation on the proposed VideoNIAH and several grounding/action tasks is limited; action recognition results are zero-shot and compared against few baselines.
- Baseline numbers are copied from original papers rather than re-run under matched settings.
Experimental setup
Datasets: WebVid InternVid ShareGPTVideo OpenVid Vript VideoChat2-IT EgoTaskQA CLEVRER LLaVA-Video-178K MovieChat PerceptionTest STAR
Benchmarks: LVBench EgoSchema LongVideoBench Video-MME MLVU VideoNIAH QVHighlights ActivityNet Something-Something V2 UCF-101
Metrics: accuracy mAP Hit@k FLOPs memory TTFT
Baselines: GPT-4o Gemini-1.5-Pro LLaVA-OneVision LLaVA-Video LLaMA-VID VideoChat-Flash LongVA LongVU Video-XL NVILA InternVL2.5 Qwen2-VL
Builds on: LLaVA-OneVision CLIP Q-Former Needle-in-a-Haystack
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.89)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.89)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.88)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.