Research map / Video / Video-language understanding
Frame-Voyager: Learning to Query Frames for Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frame-Voyager trains a lightweight plug-in module that ranks T-frame combinations of a video for a query, using ranking labels derived from a pre-trained Video-LLM's prediction losses over all combinations, improving VILA and LLaVA-OneVision on video QA.
Problem
Video-LLMs have limited input token budgets, so they must select a small subset of frames; uniform sampling and individual text-frame matching ignore information density, query complexity, and frame-to-frame interactions.
Approach
Treat frame selection as learning to rank frame combinations (not individual frames): labels come from a reference Video-LLM's language-modeling loss over all T-frame combinations of M sampled frames, and a lightweight reward model (reusing the LLM's bottom layers) is trained with a pairwise ranking loss so that combinations with lower loss get higher reward.
Key results
- On Video-MME (no subtitles), VILA-8B with Frame-Voyager improves from 47.5 to 50.5 (+3.0), and VILA-40B from 58.3 to 60.0.
- Using 8 frames selected from 128 candidates, Frame-Voyager on VILA-8B surpasses LongVILA with 128 and 256 frames (49.2 and 50.5 on Video-MME).
- On LLaVA-OneVision-7B, Frame-Voyager raises Video-MME (no subtitles) from 53.3 to 57.5 and NextQA from 72.5 to 73.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Frame-Voyager to VILA-8B | VILA-8B · Video-MME overall, no subtitles | accuracy: 47.5 -> 50.5 | Helped |
| Add Frame-Voyager to VILA-8B | VILA-8B · Video-MME long videos, no subtitles | accuracy: 40.3 -> 43.9 | Helped |
| Add Frame-Voyager to VILA-40B | VILA-40B · NextQA | accuracy: 62.9 -> 67.3 | Helped |
| Replace uniform sampling with Frame-Voyager query-conditioned frame combinations | VILA-8B · Video-MME, no subtitles | accuracy: 47.5 -> 50.5 | Helped |
| Add data filtering during training-data construction (ablation) | VideoChatGPT training data for Frame-Voyager · Video-MME, no subtitles | accuracy: 48.3 -> 49.1 | Helped |
| Train with reward ranking rather than predicting only the top-ranked combination (ablation) | Frame-Voyager · Video-MME, no subtitles | accuracy: 48.9 -> 50.5 | Helped |
| Use K=4 ranked combinations rather than K=2 (ablation) | Frame-Voyager ranking-loss training · Video-MME, no subtitles | accuracy: 49.4 -> 50.5 | Helped |
| Use K=4 ranked combinations rather than K=8 (ablation) | Frame-Voyager ranking-loss training · Video-MME, no subtitles | accuracy: 49.7 -> 50.5 | Helped |
| Expand the candidate-frame pool from 8 to 128 while selecting 8 frames (ablation) | Frame-Voyager · Video-MME, no subtitles | accuracy: 47.5 -> 50.5 | Helped |
| Use dynamic rather than fixed frame count (ablation) | Frame-Voyager on VILA-8B · Video-MME, no subtitles | accuracy: 50.5 -> 49.6 | Hurt |
| Use dynamic rather than fixed frame count (ablation) | Frame-Voyager on LLaVA-OneVision-7B · Video-MME, no subtitles | accuracy: 57.5 -> 59.1 | Mixed |
| Prune redundant combinations and CLIP-rank irrelevant frames during data collection (ablation) | Frame-Voyager data collection · Video-MME | accuracy: 50.5 -> 50.5 | No clear effect |
Surprising findings
- Rule-based shot boundary and optical-flow frame selection methods underperform uniform sampling on Video-MME.
- Training on the top-1 ranked combination only (no pairwise ranking) performs worse (48.9) than the pairwise ranking objective (50.5).
Limitations
- Ranking labels generated only with VILA-8B as reference model due to resource constraints.
- Limited to short-video training combinations (e.g., C(16,2), C(32,4)); longer-video combinations not explored.
- Reusing frozen backbone LLM layers may not be optimal; the plug-in design is simple and gains marginal on EgoSchema.
Experimental setup
Datasets: NextQA VideoChatGPT Video-MME MLVU ActivityNet-QA MVBench STAR EgoSchema
Benchmarks: Video-MME MLVU NextQA ActivityNet-QA MVBench STAR EgoSchema
Metrics: accuracy
Baselines: uniform sampling CLIP text-frame matching SigLIP InternViT-6B VILA-Embedding SeViLA TempGQA Katna LongVILA VideoLLaMA2
Builds on: VILA LLaVA-OneVision reward ranking loss
Closest papers by meaning
- M-LLM Based Video Frame Selection for Efficient Video Understanding (CVPR 2025 · similarity 0.92)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.89)
- Flexible Frame Selection for Efficient Video Reasoning (CVPR 2025 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.