Research map / Video / Video-language understanding
Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rLiVS, a training-free, Video-LLM-agnostic streaming video QA method that keeps the ~6% of visual tokens with highest LLM caption-attention, recurrently reuses them in a FIFO memory, and answers questions by retrieving captions with MMR.
Problem
Streaming video question answering over hour-long videos is hard for Video-LLMs because full-video processing is memory- and compute-intensive, KV-cache storage grows large, and clip-wise caption-only methods lose cross-clip continuity.
Approach
Select visual tokens per short clip using the LLM's own cross-attention from generated caption tokens (averaged over a few layers and heads), pass these selected tokens forward as recurrent memory for the next clip, and answer queries by retrieving captions (not visual tokens) with maximal marginal relevance.
Key results
- On RVS-Movie with LLaVA-OV 7B, rLiVS reaches 57.7% accuracy vs 54.4% for ReKV, with latency 1.9s vs 2.7s and VRAM 25GB vs 36GB.
- On MovieChat, rLiVS achieves 78.0% accuracy vs 67.6% for Goldfish (prior best listed).
- On NextQA-valset, attention-based selection keeping 6% of visual tokens reaches 77.0% vs 78.6% for the full model and 75.5% for uniform sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add recurrent past-token context (ablation) | rLiVS · RVS-Ego | accuracy: 62.5 -> 65.3 | Helped |
| add recurrent past-token context (ablation) | rLiVS · RVS-Movie | accuracy: 53.7 -> 57.7 | Helped |
| add recurrent past-token context (ablation) | rLiVS · MovieChat | accuracy: 74.1 -> 78.0 | Helped |
| retrieve captions only instead of selected visual tokens (ablation) | rLiVS · RVS-Ego | accuracy: 58.2 -> 65.1 | Helped |
| retrieve captions only instead of selected visual tokens (ablation) | rLiVS · RVS-Movie | accuracy: 48.4 -> 57.7 | Helped |
| use captions only rather than captions plus selected visual tokens for answering (ablation) | rLiVS · RVS-Ego | accuracy: 63.0 -> 65.1 | Helped |
| use captions only rather than captions plus selected visual tokens for answering (ablation) | rLiVS · RVS-Movie | accuracy: 54.3 -> 57.7 | Helped |
| replace uniform sampling with attention-based visual token selection (ablation) | rLiVS · RVS-Ego | accuracy: 64.2 -> 65.1 | Helped |
| replace uniform sampling with attention-based visual token selection (ablation) | rLiVS · RVS-Movie | accuracy: 56.0 -> 57.7 | Helped |
| replace uniform sampling with attention-based selection at 6% retention (ablation) | short-clip token selection · NextQA-valset | accuracy: 75.5 -> 77.0 | Helped |
| replace mean pooling with attention-based selection at 6% retention (ablation) | short-clip token selection · NextQA-valset | accuracy: 70.7 -> 77.0 | Helped |
| increase attention-selected visual token retention from 6% to 12% (ablation) | short-clip token selection · NextQA-valset | accuracy: 77.0 -> 78.4 | Helped |
Surprising findings
- Retrieving selected visual tokens instead of captions performs worse (48.4% on RVS-Movie) than captions (57.7%), attributed to mismatch between short-clip Video-LLM training and long streams.
- Selecting 25% of tokens on NextQA slightly outperforms the full-token baseline (79.0 vs 78.6).
Limitations
- Attending only to selected tokens during clip processing may overlook fine-grained details and harm continuity.
- Captions may be redundant across clips, affecting retrieval; method inherits limitations of pretrained backbones.
- Spatial positional information is lost when reinserting selected tokens, left for future work.
Experimental setup
Datasets: NextQA-valset RVS-Ego RVS-Movie MovieChat VStream-QA offline (VS-Ego, VS-Movie) CG-Bench MLVU validation
Benchmarks: RVS-Ego RVS-Movie VS-Ego VS-Movie MovieChat CG-Bench NextQA-valset MLVU
Metrics: accuracy score (GPT-3.5 0-5) end-to-end latency VRAM KV-cache size
Baselines: VideoChatGPT MovieChat Chat-UniVi LLaMA-VID VideoScan Flash-VStream-7B Goldfish ReKV Video-XL Qwen2.5-VL
Builds on: LLaVA-OneVision ReKV Goldfish Maximal Marginal Relevance
Closest papers by meaning
- M-LLM Based Video Frame Selection for Efficient Video Understanding (CVPR 2025 · similarity 0.91)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.90)
- FlexSelect: Flexible Token Selection for Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.89)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.