Research map / Video / Video-language understanding
BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOLT, a training-free test-time frame selection method (inverse transform sampling over CLIP/SigLIP query-frame similarity) that boosts off-the-shelf video VLMs on long-form video QA.
Problem
Large video-language models are limited by context windows, so uniform frame sampling wastes tokens on query-irrelevant content in long, noisy videos; prior adaptive selection methods require costly retraining.
Approach
Use query-frame similarity from CLIP/SigLIP to build a sharpened cumulative distribution over frames and sample frames by inverse transform sampling at inference time, with no VLM retraining; also introduces a multi-source retrieval evaluation that concatenates videos to simulate noisy long videos.
Key results
- On Video-MME, LLaVA-OneVision with 8 frames improves from 53.8% to 56.1% with BOLT (ITS).
- On MLVU with 8 frames, accuracy rises from 58.9% to 63.4% with BOLT.
- Under multi-source retrieval evaluation with 16 frames, LLaVA-OneVision on Video-MME improves from 47.8% to 53.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply inverse transform sampling for query-guided frame selection | LLaVA-OneVision-7B · Video-MME (w.o. sub.) | accuracy: 53.8 -> 56.1 | Helped |
| Apply inverse transform sampling for query-guided frame selection | LLaVA-OneVision-7B · MLVU | accuracy: 58.9 -> 63.4 | Helped |
| Apply inverse transform sampling for query-guided frame selection | LLaVA-OneVision-7B · NextQA | accuracy: 77.4 -> 77.4 | No clear effect |
| Apply inverse transform sampling for frame selection | LLaVA-OneVision-7B · Video-MME multi-source retrieval | accuracy: 47.78 -> 53.37 | Helped |
| Replace uniform sampling with top-K similarity selection (ablation) | LLaVA-OneVision-7B · Video-MME standard | accuracy: +0.25 | Helped |
| Replace uniform sampling with watershed grouping (ablation) | LLaVA-OneVision-7B · Video-MME standard | accuracy: +0.38 | Helped |
| Replace uniform sampling with inverse transform sampling (ablation) | LLaVA-OneVision-7B · Video-MME standard | accuracy: +0.93 | Helped |
| Shuffle uniformly sampled frames (ablation) | LLaVA-OneVision-7B · Video-MME standard | accuracy: -0.60 | Hurt |
| Replace uniform sampling with random sampling (ablation) | LLaVA-OneVision-7B · Video-MME standard | accuracy: -1.70 | Hurt |
| Use CLIP rather than caption similarity for frame-query scoring (ablation) | LLaVA-OneVision-7B with ITS · Video-MME | accuracy: 55.47 -> 57.78 | Helped |
| Average CLIP and SigLIP similarity scores (ablation) | LLaVA-OneVision-7B with ITS · Video-MME | accuracy: 57.78 -> 57.83 | Helped |
| Set ITS sharpness α between 2 and 3.5 rather than use uniform sampling (ablation) | LLaVA-OneVision-7B with ITS · Video-MME, MLVU, LongVideoBench | accuracy: 56.90 -> 58.37; 61.24 -> 65.08; 55.72 -> 58.04 | Helped |
Surprising findings
- A blind test without any visual input reaches about 41% on Video-MME, indicating VLMs rely heavily on general knowledge.
- Uniform sampling with 64 frames still degrades substantially under multi-source retrieval (52.1% vs 56.9% standard), showing background noise harms VLMs.
Limitations
- Effectiveness depends on accuracy of query-frame similarity, which may not reflect relevance for reasoning-heavy queries.
- Gains on NextQA are negligible, likely due to short videos.
- Evaluated mainly on single-turn multiple-choice VQA; multi-round VQA not explored.
Experimental setup
Datasets: MultiHop-EgoQA Video-MME MLVU EgoSchema LongVideoBench NextQA
Benchmarks: Video-MME MLVU EgoSchema LongVideoBench NextQA
Metrics: accuracy average sentence similarity
Baselines: Uniform sampling Top-K selection Watershed grouping Random sampling Frame-Voyager LongVU LLaVA-OneVision
Builds on: CLIP SigLIP LLaVA-OneVision Frame-Voyager
Closest papers by meaning
- Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.91)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.89)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.