Research map / Video / Video-language understanding
Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReKV, a training-free framework that adds sliding-window attention, offloaded video KV-cache storage, and query-time retrieval of relevant KV-caches to existing Video-LLMs for streaming video QA.
Problem
Streaming video question-answering requires answering queries about any point in a long video stream with low latency and bounded memory, which existing Video-LLMs handle poorly because they must re-encode the whole video for each question and lose detail through sparse sampling or token compression
Approach
Encode the video stream incrementally with sliding-window attention, offload all video KV-caches to RAM/disk, and at question time retrieve only the top query-relevant frame blocks' KV-caches (via an external SigLIP retriever or the Video-LLM's own attention keys) to serve as context for answering, with video encoding and QA decoupled across processes/GPUs.
Key results
- LLaVA-OV-7B+ReKV improves over uniform sampling on MLVU dev (64.7 to 68.5), QAEgo4D test (52.8 to 56.0), EgoSchema (59.8 to 60.7), and ActivityNet-QA (56.6 to 60.4).
- On RVS-Ego and RVS-Movie streaming QA, LLaVA-OV-7B with internal retrieval reaches 63.7 and 54.4 accuracy, above Flash-VStream-7B (57.3 and 53.1).
- Internal retrieval on LLaVA-OV-7B lowers latency from 5.8s (external) to 3.3s and GPU memory from 55GB to 38GB while encoding at 11 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add internal KV-cache retrieval | LLaVA-OV-7B · MLVU dev-mc | accuracy: 64.7 -> 68.5 | Helped |
| Add internal KV-cache retrieval | LLaVA-OV-7B · QaEgo4D test-mc | accuracy: 52.8 -> 56.0 | Helped |
| Add internal KV-cache retrieval | LLaVA-OV-7B · EgoSchema | accuracy: 59.8 -> 60.7 | Helped |
| Add internal KV-cache retrieval | LLaVA-OV-7B · ActivityNet-QA | accuracy: 56.6 -> 60.4 | Helped |
| Add internal KV-cache retrieval | LLaVA-OV-7B · RVS-Ego and RVS-Movie | accuracy: 56.2 -> 63.7; 43.0 -> 54.4 | Helped |
| Replace uniform sampling with external retrieval (ablation) | LLaVA-OV-7B · QaEgo4D test-mc | accuracy: 53.0 -> 54.2 | Helped |
| Replace uniform sampling with internal retrieval (ablation) | LLaVA-OV-7B · QaEgo4D test-mc | accuracy: 53.0 -> 56.0 | Helped |
| Replace external retrieval with internal retrieval (ablation) | LLaVA-OV-7B · QaEgo4D test-mc | accuracy: 54.2 -> 56.0 | Helped |
| Increase the number of retrieved frames (ablation) | ReKV with LLaVA-OV-7B · QaEgo4D and MLVU | accuracy | Helped |
| Increase the number of retrieved frames (ablation) | ReKV with LLaVA-OV-7B · MLVU | accuracy | No clear effect |
| Increase retrieval block size (ablation) | ReKV with LLaVA-OV-7B · MLVU | accuracy | Hurt |
| Increase retrieval block size (ablation) | ReKV with LLaVA-OV-7B · QaEgo4D | accuracy | No clear effect |
Surprising findings
- Applying standard RoPE positions to retrieved KV-caches works better than preserving original positions or assigning a static position (Inf-LLM style), which the authors attribute to the importance of temporal information.
- On MLVU, retrieving more dispersed blocks (larger block size) hurts accuracy, while QAEgo4D is stable across block sizes.
Limitations
- KV-cache size grows with video length, potentially unsustainable for extremely long streams such as surveillance without compression.
- Constant block size for retrieval can break video continuity; retrieves a fixed number of frames rather than adapting to the query.
- StreamingVQA benchmarks are few, and the method's gains depend on the base Video-LLM's capability.
Experimental setup
Datasets: MLVU QAEgo4D EgoSchema ActivityNet-QA RVS-Ego RVS-Movie CGBench
Benchmarks: MLVU dev-mc QAEgo4D test-mc EgoSchema ActivityNet-QA RVS-Ego RVS-Movie CGBench mc
Metrics: accuracy open-ended score (1-5) latency GPU memory TFLOPs per QA recall
Baselines: Uniform Sampling Flash-VStream VideoStreaming GPT-4o Gemini-1.5-Pro LongVA Video-LLaVA LLaVA-OV
Builds on: LM-Infinite Inf-LLM StreamingLLM LLaVA-OneVision
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.88)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
- BIMBA: Selective-Scan Compression for Long-Range Video Question Answering (CVPR 2025 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.87)
- Koala: Key Frame-Conditioned Long Video-LLM (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.