Research map / Video / Video-language understanding
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video-RAG is a training-free, plug-and-play pipeline that adds retrieved OCR, ASR, and object-detection auxiliary texts to any open-source LVLM to improve long video question answering.
Problem
Large video-language models struggle with long videos due to limited context; fine-tuning long-context LVLMs is data and GPU intensive, and GPT-based agent methods are slow and depend on proprietary models.
Approach
Replace extra visual tokens with query-filtered, visually aligned auxiliary texts (OCR, ASR, object detection via open-source tools) retrieved with RAG (Contriever + FAISS) and feed them alongside frames and the query into an existing LVLM, with the LVLM itself decomposing the query into retrieval requests.
Key results
- LLaVA-Video-72B with Video-RAG reaches 77.4% on Video-MME overall vs 77.2% for GPT-4o with subtitles-free comparison baseline in the paper's table.
- Average gain of 2.8% on Video-MME across seven LVLMs with about 2.0K auxiliary tokens added.
- LLaVA-Video-72B + Video-RAG achieves 65.4% on LongVideoBench validation, 1.4% above Gemini-1.5-Pro (64.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Video-RAG to Long-LLaVA-7B | Long-LLaVA-7B · Video-MME Overall | accuracy: 57.8 -> 62.6 | Helped |
| Add Video-RAG to LLaVA-Video-72B | LLaVA-Video-72B · Video-MME Overall | accuracy: 75.9 -> 77.4 | Helped |
| Add Video-RAG to LLaVA-Video-7B | LLaVA-Video-7B · MLVU Overall | accuracy: 70.8 -> 72.4 | Helped |
| Add Video-RAG to LLaVA-Video-72B | LLaVA-Video-72B · MLVU Overall | accuracy: 73.1 -> 73.8 | Helped |
| Add Video-RAG to LLaVA-Video-7B | LLaVA-Video-7B · LongVideoBench validation Overall | accuracy: 56.6 -> 58.7 | Helped |
| Add Video-RAG to LLaVA-Video-72B | LLaVA-Video-72B · LongVideoBench validation Overall | accuracy: 61.9 -> 65.4 | Helped |
| Add retrieved auxiliary texts (RAG) to DET, OCR, and ASR (ablation) | Long-LLaVA-7B · Video-MME Overall | accuracy: 59.8 -> 62.1 | Helped |
| Add ASR auxiliary text without RAG (ablation) | Long-LLaVA-7B · Video-MME Overall | accuracy: 52.0 -> 57.4 | Helped |
| Add DET, OCR, and ASR without RAG (ablation) | LLaVA-Video-7B · MLVU Overall | accuracy: 72.4 -> 70.3 | Hurt |
| Add RAG, DET, OCR, and ASR (ablation) | Long-LLaVA-7B · Video-MME Overall | accuracy: 52.0 -> 62.1 | Helped |
| Replace RAG retrieval with random auxiliary text (ablation) | Long-LLaVA-7B · Video-MME Overall | accuracy: 59.1 vs 62.1 | Hurt |
| Set retrieval threshold to 0.5 instead of 0.3 (ablation) | Long-LLaVA-7B + Video-RAG · Video-MME Overall | accuracy: 62.1 -> 56.1 | Hurt |
Surprising findings
- LongVA performance declines when raising frame sampling from 128 to 384 on Video-MME, suggesting more frames add redundancy.
- Adding ASR auxiliary text hurts VNBench overall performance, since subtitles are not relevant to that synthetic task.
Limitations
- Performance is bounded by the quality of the chosen external visual tools (OCR, ASR, detection) and lacks adaptation.
- Object detection and ASR add substantial preprocessing time (e.g., ASR up to 42 min on the full set for some settings).
Experimental setup
Datasets: Video-MME MLVU LongVideoBench VNBench
Benchmarks: Video-MME MLVU LongVideoBench VNBench
Metrics: accuracy
Baselines: GPT-4o Gemini-1.5-Pro LLaVA-Video LongVA Long-LLaVA VideoAgent Qwen2-VL
Builds on: Retrieval-Augmented Generation Contriever FAISS Whisper EasyOCR APE
Closest papers by meaning
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.93)
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.92)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.89)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.