Research map / Video / Video-language understanding
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexSelect, a training-free token ranking method using cross-modal attention from a reference layer of a VideoLLM, plus a rank-supervised lightweight selector, to prune visual tokens for efficient long-video understanding.
Problem
Long-form video understanding with VideoLLMs is limited by the huge number of visual tokens, causing prohibitive compute and memory costs and limited context length.
Approach
Identify an intermediate "reference layer" in VideoLLMs whose cross-modal attention faithfully ranks query-relevant visual tokens (via needle-in-haystack Recall@K), use its scores to select top tokens per frame set training-free, and train a small 0.5B transformer selector with a differentiable Spearman rank loss to mimic those rankings.
Key results
- FlexSelect on LLaVA-Video-7B raises VideoMME overall from 64.4 to 68.9 and LVBench from 43.1 to 52.9.
- On LLaVA-Video-7B with 512 frames, response time drops from 38.24 s to 4.23 s (about 9x speedup) using FlexSelect-Lite.
- Qwen2.5-VL-72B with FlexSelect reaches 56.6 on LVBench (from 47.3) and 74.4 on VideoMME overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FlexSelect token selection | LLaVA-Video-7B · VideoMME Overall | accuracy: 64.4 -> 68.9 | Helped |
| add FlexSelect token selection | LLaVA-Video-7B · LVBench Test | score: 43.1 -> 52.9 | Helped |
| add FlexSelect token selection | InternVL2.5-8B · VideoMME Long | accuracy: 52.8 -> 58.1 | Helped |
| add FlexSelect token selection | Qwen2.5-VL-7B · LVBench Test | score: 45.3 -> 51.2 | Helped |
| replace reference-layer scoring with the lightweight rank-supervised selector | LLaVA-Video-7B FlexSelect · VideoMME Overall | accuracy: 68.9 -> 68.3 | Hurt |
| use 512 rather than 64 input frames (ablation) | LLaVA-Video-7B FlexSelect · VideoMME | accuracy: 65.2 -> 68.9 | Helped |
| increase input frames from 512 to 1,024 (ablation) | LLaVA-Video-7B FlexSelect · VideoMME | accuracy: 68.9 -> 68.1 | Hurt |
| increase max selected tokens from 1,680 to 6,720 (ablation) | LLaVA-Video-7B FlexSelect · VideoMME | accuracy: 67.1 -> 68.9 | Helped |
| increase max selected tokens from 6,720 to 13,440 (ablation) | LLaVA-Video-7B FlexSelect · VideoMME | accuracy: 68.9 -> 68.1 | Hurt |
| train the selector on 5% rather than no training (ablation) | LLaVA-Video-7B FlexSelect-Lite · VideoMME | accuracy: 61.9 -> 63.9 | Helped |
| increase selector training data from 5% to 10% (ablation) | LLaVA-Video-7B FlexSelect-Lite · VideoMME | accuracy: 63.9 -> 63.5 | Hurt |
| use mixed instruction types instead of detail captions (ablation) | LLaVA-Video-7B FlexSelect-Lite · VideoMME | accuracy: 63.5 -> 63.9 | Helped |
Surprising findings
- Attention from intermediate layers, not early or final layers, best recovers needle tokens; early layers and very deep layers both perform poorly.
- Scaling the lightweight token selector from 0.5B to 3B gives only marginal accuracy gains while tripling response time.
Limitations
- Performance is capped by the host VideoLLM's native capabilities since the base model is not trained.
- FlexSelect-Lite typically underperforms direct reference-layer selection.
- Selector training is excluded for 72B models due to memory constraints.
Experimental setup
Datasets: LLaVA-Video-178K VideoMME MLVU LongVideoBench LVBench V-NIAH (from LongVA)
Benchmarks: VideoMME MLVU LongVideoBench LVBench
Metrics: accuracy Recall@K response time (time to first token) Spearman correlation
Baselines: LLaVA-Video InternVL2.5 Qwen2.5-VL GPT-4o Gemini-1.5-Pro SF-LLaVA-1.5 TPO Quato ViLAMP FastV PyramidDrop SparseVLM
Builds on: FastV SparseVLM PyramidDrop LongVA LLaVA-OneVision
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.89)
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.88)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.88)
- Flexible Frame Selection for Efficient Video Reasoning (CVPR 2025 · similarity 0.88)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.