Research map / Video / Video-language understanding
M-LLM Based Video Frame Selection for Efficient Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a lightweight Qwen2.5-1.5B-based question-aware frame selector, supervised by spatial and temporal pseudo-labels, to pick key frames for frozen video M-LLMs, improving QA with fewer frames.
Problem
Uniform frame sampling in video M-LLMs wastes context on irrelevant or redundant frames and misses key moments, especially in long videos, while there is no frame-level importance supervision for video QA.
Approach
A small LLM with a learnable score query predicts per-frame importance for all frames in one pass from heavily pooled tokens (3x3 per frame); pseudo-labels combine spatial single-frame scores from an M-LLM (CoT True/False probability) and temporal top-k selections by a text-only LLM over frame captions; NMS-greedy sampling picks the final frames for a frozen video M-LLM.
Key results
- LLaVA-NeXT-Video 34B + selector reaches 60.2 on ActivityNet-QA vs 58.8 uniform; PLLaVA 34B + selector 62.3 (vs 60.9).
- Qwen2-VL 7B + selector improves EgoSchema from 64.6 to 65.9 and NExT-QA from 77.6 to 78.4 (reported in tables); on VideoMME average rises 58.1 to 58.7.
- On LongVideoBench, Qwen2-VL 7B with 16 selected frames (56.5) beats 32 uniform frames (53.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the frame selector instead of uniform sampling | LLaVA-NeXT-Video 7B · ActivityNet-QA | accuracy: 53.5 -> 55.1 | Helped |
| Use the frame selector instead of uniform sampling | LLaVA-NeXT-Video 7B · NExT-QA | accuracy: 62.4 -> 63.4 | Helped |
| Use the frame selector | Qwen2-VL · EgoSchema | accuracy: 64.6 -> 65.9 | Helped |
| Combine spatial and temporal pseudo labels rather than spatial labels alone (ablation) | LLaVA-NeXT-Video 7B frame selection · ActivityNet-QA | accuracy: 54.2 -> 55.5 | Helped |
| Combine spatial and temporal pseudo labels rather than spatial labels alone (ablation) | LLaVA-NeXT-Video 7B frame selection · NExT-QA | accuracy: 63.6 -> 63.9 | Helped |
| Use CLIP similarity scores for frame selection instead of uniform sampling (ablation) | LLaVA-NeXT-Video 7B · ActivityNet-QA | accuracy: 53.5 -> 53.7 | Helped |
| Use CLIP similarity scores for frame selection instead of uniform sampling (ablation) | LLaVA-NeXT-Video 7B · NExT-QA | accuracy: 62.4 -> 62.2 | Hurt |
| Increase selector input from 1 to 9 tokens per frame (ablation) | LLaVA-NeXT-Video 7B frame selector · ActivityNet-QA | accuracy: 53.2 -> 55.1 | Helped |
| Increase selector backbone from 0.5B to 1.5B (ablation) | LLaVA-NeXT-Video 7B frame selector · NExT-QA | accuracy: 62.8 -> 63.4 | Helped |
| Use 7B rather than 1.5B selector backbone (ablation) | LLaVA-NeXT-Video 7B frame selector · NExT-QA | accuracy: 63.4 -> 64.0 | Helped |
| Select 8 frames from 128 candidates rather than uniform-sample 8 frames (ablation) | LLaVA-NeXT-Video 34B · NExT-QA | accuracy: 68.1 -> 69.3 | Helped |
| Select 8 frames from 128 candidates rather than uniform-sample 8 frames (ablation) | LLaVA-NeXT-Video 34B · NExT-QA | speed: 0.92 -> 1.12 | Hurt |
Surprising findings
- A small 0.5B selector already yields most of the gain, and 9 tokens per frame suffice, as more tokens per frame bring little improvement.
- Image-trained M-LLMs (Qwen2-VL) produced better temporal pseudo-labels than video-trained LLaVA-NeXT-Video.
Limitations
- Temporal pseudo-labels rely on a two-stage caption-based process that may lose information or hallucinate.
- On QVHighlights moment retrieval the selector underperforms SeViLA (43.9 vs 54.5 R1@5).
- Pseudo-labels depend on proprietary GPT-4o mini and sizable M-LLM inference, making label generation costly.
Experimental setup
Datasets: VideoChat2 training data TimeIT LLaVA-Video-178K ActivityNet-QA NExT-QA EgoSchema VideoMME LongVideoBench QVHighlights
Benchmarks: ActivityNet-QA NExT-QA EgoSchema VideoMME LongVideoBench QVHighlights
Metrics: accuracy correctness score R1@5 R1@7 inference speed
Baselines: Uniform sampling CLIP similarity frame scores SeViLA pseudo labels PLLaVA LLaVA-NeXT-Video Qwen2-VL Idefics2 Tarsier SlowFast-LLaVA
Builds on: SeViLA LLaVA-NeXT-Video PLLaVA Qwen2-VL
Closest papers by meaning
- Flexible Frame Selection for Efficient Video Reasoning (CVPR 2025 · similarity 0.92)
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.92)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.91)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.89)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.89)
- Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.