Research map / Video / Video-language understanding
SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SALOVA, a video-LMM that retrieves query-relevant video segments via a segment retrieval router and a spatio-temporal connector, trained on the new 87.8K-video SceneWalk dataset of segment-level dense captions.
Problem
Long untrimmed videos exceed the context length and memory budget of video LMMs, so sparse frame sampling loses critical information and degrades response relevance.
Approach
Treat long-video understanding as retrieval: segment videos by scene changes, score segments against the query with a learned router trained on video-text correspondence scores, and feed only the top-K segments (focus pathway) plus segment-level routing tokens (fast pathway) into the LLM.
Key results
- SALOVA-7B reaches 53.1 overall on Video-MME (w/o subtitles), versus 52.6 for LongVA-7B, with 44.6 on LongVideoBench val.
- SALOVA-3B improves Video-MME overall from 42.5 (8 frames) and 43.6 (16 frames) to 45.3 with 1 FPS sampling plus the segment retrieval router.
- Removing the FocusFast local-global representation drops SALOVA-3B's Video-MME overall from 45.3 to 36.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 1 FPS sampling instead of 16-frame sampling (ablation) | SALOVA-3B without SR-Router · Video-MME Overall | accuracy: 43.6 -> 45.3 | Helped |
| Use 1 FPS sampling instead of 8-frame sampling (ablation) | SALOVA-3B without SR-Router · Video-MME Overall | accuracy: 42.5 -> 45.3 | Helped |
| Add long-video knowledge injection training (Stage 1.5) (ablation) | SALOVA-3B · Video-MME Overall | accuracy: 43.6 -> 45.3 | Helped |
| Add FocusFast local-global video representation (ablation) | SALOVA-3B · Video-MME Overall | accuracy: 36.9 -> 45.3 | Helped |
| Use SALOVA-7B instead of LongVA | Video-MME · Video-MME Overall | accuracy: 52.6 -> 53.1 | Helped |
| Use SALOVA-7B instead of VideoLLaMA2 | Video understanding models · ActivityNetQA (test) | accuracy: 50.2 -> 53.6 | Helped |
| Use SALOVA-7B instead of VideoLLaMA2 | Video understanding models · VideoChatGPT (test) | score: 3.3 -> 3.5 | Helped |
| Use SALOVA-7B instead of VideoLLaMA2 | Video understanding models · MVBench (test) | accuracy: 54.6 -> 53.5 | Hurt |
Surprising findings
- On the general benchmarks, SALOVA-3.8B scores 46.4 on MVBench, well below VideoChat2 and VideoLLaMA2, and the paper concedes simpler models may suffice for shorter videos.
Limitations
- Architecture may be unnecessary for shorter videos where sparse sampling suffices, as the authors acknowledge.
- General video benchmark results are competitive but not consistently best, e.g. MVBench 46.4 for SALOVA-3.8B versus 54.6 for VideoLLaMA2.
Experimental setup
Datasets: SceneWalk CC3M WebVid 2.5M LLaVA-Video-178K NeXT-QA ActivityNetQA PerceptionTest Video-MME LongVideoBench VideoChatGPT MVBench
Benchmarks: Video-MME LongVideoBench ActivityNetQA VideoChatGPT MVBench V-NIAH
Metrics: accuracy score
Baselines: LongVA VideoLLaMA2 SliME ShareGPT4Video VideoChat2 Video-LLaVA GPT-4o Gemini 1.5 Pro
Builds on: LLaVA Perceiver Resampler SlowFast LanguageBind
Closest papers by meaning
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.89)
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.87)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.86)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.