Research map / Video / Video-language understanding
Flexible Frame Selection for Efficient Video Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FFS, a lightweight learnable frame selector with a [CTRL] register token that adaptively picks between k_low and k_high frames for video-language QA, trained end-to-end through a downstream VLM via Gumbel top-k straight-through estimation.
Problem
Uniform frame sampling for video-language models ignores question and video context, and prior learned selectors either use fixed k or are expensive VLM/LLM-based, so accuracy and efficiency trade off poorly on long-video QA.
Approach
A learnable [CTRL] register token appended to the selector's multimodal encoder lets a fixed top-k selection operator produce a variable number of frames within user-set bounds [k-c, k], with no VLM/LLM calls in the selector and end-to-end training via a Gumbel top-k straight-through estimator.
Key results
- On NExT-QA, EVA-V-L + FFS (2.3 frames average from [1,4]) reaches 62.2% accuracy vs 61.6% for uniform 32-frame EVA-V-L with 13.5x fewer downstream frames/FLOPs.
- On EgoSchema, OpenLLoVi (Gemma) + FFS (16.8 frames from [4,32]) reaches 55.6% test accuracy vs 53.6% for uniform 16 frames and 55.4% for uniform 180 frames.
- On IntentQA, EVA-V-L + FFS (2.4 frames from [1,4]) reaches 64.2% overall vs 63.6% for uniform 32-frame EVA-V-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive FFS frame selection | EVA-V-L · IntentQA test | accuracy (Overall): 63.6 -> 64.2 | Helped |
| Add adaptive FFS frame selection | EVA-V-L · NExT-QA validation | accuracy (All): 61.6 -> 62.2 | Helped |
| Add adaptive FFS frame selection | EVA-V-L · NExT-QA validation | FLOPs: 246.8T -> 18.2T | Helped |
| Add adaptive FFS frame selection | OpenLLoVi (Gemma) · NExT-QA validation | accuracy (All): 66.5 -> 66.7 | Helped |
| Add adaptive FFS frame selection | OpenLLoVi (Gemma) · EgoSchema test | accuracy: 55.4 -> 55.6 | Helped |
| Replace uniform 32-frame sampling with FFS | EVA-V-L · NExT-QA validation | average downstream frames: 32 -> 2.3 from 32 | Helped |
| Remove [CTRL] tokens and use fixed top-k selection (ablation) | FFS · NExT-QA validation subset | accuracy | Hurt |
| Use a fixed top-1 policy instead of flexible frame selection (ablation) | FFS · NExT-QA validation subset | accuracy | Hurt |
Surprising findings
- The selector's adaptive selection reaches or exceeds accuracy of uniform sampling over 32 or 180 frames while using only a few frames on average.
Limitations
- Relies on downstream VLM supervision and auxiliary top-k losses, with the selector's performance tied to the chosen k_low/k_high bounds.
- Evaluated on multiple-choice videoQA benchmarks only; generalization to open-ended tasks is not shown in the main text.
Experimental setup
Datasets: IntentQA NExT-QA EgoSchema
Benchmarks: IntentQA NExT-QA EgoSchema
Metrics: accuracy frames processed FLOPs
Baselines: Uniform frame sampling Fixed top-k selection Temp[ATP] SeViLa VideoAgent (GPT-4) LLoVi (GPT-4) CaVIR MoReVQA
Builds on: Gumbel-Max straight-through estimator SeViLa VideoAgent LLoVi EVA-CLIP
Closest papers by meaning
- M-LLM Based Video Frame Selection for Efficient Video Understanding (CVPR 2025 · similarity 0.92)
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.89)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- FlexSelect: Flexible Token Selection for Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.