Research map / Video / Video-language understanding
Adaptive Keyframe Sampling for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Keyframe Sampling (AKS), a plug-and-play keyframe selector that balances prompt-frame relevance (BLIP image-text matching) and temporal coverage (recursive binning), improving frozen video MLLMs on long video QA.
Problem
Video MLLMs can only ingest a limited number of visual tokens, so uniform or ad hoc frame sampling for long videos often drops the frames needed to answer the question.
Approach
Formulate keyframe selection as maximizing prompt-frame relevance plus temporal coverage under a fixed frame budget, and approximate it with an adaptive recursive judge-and-split procedure (ADA) that interpolates between top-score sampling and binned uniform-coverage sampling, without tuning the MLLM.
Key results
- With AKS, LLaVA-Video-7B reaches 62.7% on LongVideoBench val (vs 58.9% uniform) and 65.3% on VideoMME (vs 64.4%), surpassing GPT-4V and Gemini-1.5-Flash on LongVideoBench.
- AKS improves Qwen2-VL-7B by 5.0 points on LongVideoBench (55.5 to 60.5) and LLaVA-OV-7B by 4.5 points (54.8 to 59.3).
- ADA sampling outperforms TOP and BIN sampling on both benchmarks, combining the strengths of each.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AKS keyframe selection | Qwen2-VL-7B · LongVideoBench val | accuracy: 55.5 -> 60.5 | Helped |
| Add AKS keyframe selection | Qwen2-VL-7B · VideoMME | accuracy: 57.6 -> 59.9 | Helped |
| Add AKS keyframe selection | LLaVA-OV-7B · LongVideoBench val | accuracy: 54.8 -> 59.3 | Helped |
| Add AKS keyframe selection | LLaVA-OV-7B · VideoMME | accuracy: 56.5 -> 58.4 | Helped |
| Add AKS keyframe selection | LLaVA-Video-7B · LongVideoBench val | accuracy: 58.9 -> 62.7 | Helped |
| Add AKS keyframe selection | LLaVA-Video-7B · VideoMME | accuracy: 64.4 -> 65.3 | Helped |
| Use adaptive sampling rather than uniform sampling (ablation) | LLaVA-Video-7B · LongVideoBench val | accuracy: 58.9 -> 62.7 | Helped |
| Use adaptive sampling rather than uniform sampling (ablation) | LLaVA-Video-7B · VideoMME | accuracy: 64.4 -> 65.3 | Helped |
| Use adaptive sampling instead of top-scoring sampling (ablation) | LLaVA-Video-7B · LongVideoBench val | accuracy: 62.4 -> 62.7 | Helped |
| Use adaptive sampling instead of top-scoring sampling (ablation) | LLaVA-Video-7B · VideoMME | accuracy: 63.7 -> 65.3 | Helped |
| Use adaptive sampling instead of binned sampling (ablation) | LLaVA-Video-7B · LongVideoBench val | accuracy: 60.2 -> 62.7 | Helped |
| Use adaptive sampling instead of binned sampling (ablation) | LLaVA-Video-7B · VideoMME | accuracy: 65.2 -> 65.3 | No clear effect |
Surprising findings
- Maximizing relevance alone (TOP sampling) can hurt on VideoMME, where multi-moment questions need temporal coverage.
- Sampling candidates at as low as 0.1 fps still beats the uniform baseline on LongVideoBench.
Limitations
- Relies on an external image-text matching model (BLIP) for relevance scores, adding computation.
- Gains on VideoMME are smaller than on LongVideoBench (e.g., +0.9 points for LLaVA-Video).
Experimental setup
Datasets: LongVideoBench VideoMME
Benchmarks: LongVideoBench VideoMME
Metrics: accuracy
Baselines: Uniform sampling (LLaVA-Video default) TOP sampling BIN sampling GPT-4V GPT-4o Gemini-1.5-Pro Qwen2-VL-7B LLaVA-OV-7B VideoLLaVA MiniCPM-V 2.6 PLLaVA VILA
Builds on: LLaVA-Video BLIP CLIP Ripley's K-function
Closest papers by meaning
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.91)
- Exploring the Design Space of Visual Context Representation in Video MLLMs (ICLR 2025 · similarity 0.90)
- Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.