Research map / Video / Video-language understanding
SLVMBench: Skill Learning from Video Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SLVMBench, a 2,261-question multiple-choice benchmark where video-LLMs must learn a skill from a tutorial video hidden in 10 min to 2 h of distractor video and apply it to a real-time question at a cutoff timestamp.
Problem
Existing benchmarks evaluate either real-time streaming perception or long-video passive comprehension, but not whether video-LLMs can acquire procedural skills from long video memory and apply them in real time.
Approach
Embed a tutorial video inside a long stream of unrelated distractor videos, then ask decision-point questions on a target video; combines knowledge-coverage-verified tutorial/target pairs, post-2025 YouTube videos, GPT-4o/Gemini-generated QA refined by humans with sub-second cutoff calibration and de-spoiled, plausible distractors.
Key results
- Gemini 3.1 Pro reaches 74.92% with the tutorial (vs 45.45% without), but drops to 59.45% when the tutorial is placed 10 min to 2 h earlier in the stream.
- GPT-5.2 and GPT-4o gains shrink to 2.21 and 2.48 points under the long-memory setting.
- video-SALMONN-S retains a 14.95-point gain in the long setting, with CB accuracy of 73.16%, while Qwen3-VL (8B) reaches only 16.09% on Conditional Branching.
Surprising findings
- PEMF, a streaming model, falls from 64.88% to 39.36% in the long setting, so streaming design alone does not guarantee episodic retention.
- Audio-capable models (Qwen2.5-Omni 7B, gain 11.2) retain tutorial gains better than the visual-only Qwen3-VL 8B (gain 1.28).
Limitations
- Tutorial-to-query gap is capped at 2 hours due to current input length limits of video-LLMs.
- Evaluation is abstracted into multiple-choice QA rather than an interactive embodied or agent environment.
- Some evaluated models' configurations are unclear; proprietary API dates and the 2025 date stated in the text are inconsistent.
Experimental setup
Datasets: SLVMBench
Benchmarks: SLVMBench
Metrics: accuracy Wilson 95% confidence interval
Baselines: Gemini 3.1 Pro GPT-5.2 GPT-4o video-SALMONN-S StreamMem PEMF video-SALMONN 2+ Qwen3-Omni Qwen2.5-Omni VideoLLaMA3 Qwen3-VL
Builds on: VideoMME LVBench StreamingBench VideoMMMU DemoICL
Closest papers by meaning
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.89)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.88)
- Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (ICLR 2025 · similarity 0.88)
- VideoWorld: Exploring Knowledge Learning from Unlabeled Videos (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.