Research map / Video / Video-language understanding
LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiveStar, a live streaming Video-LLM trained with streaming causal attention masks and decoded via a single-pass perplexity-verification response-silence scheme, plus the OmniStar dataset and benchmark.
Problem
Online Video-LLMs struggle to process continuous frame-by-frame video streams and decide when to respond, relying heavily on EOS tokens that hurt alignment and training; existing benchmarks cover narrow scenarios and single tasks.
Approach
Replace EOS-based silence modeling with streaming causal attention masks (SCAM) for interleaved frame-caption training and a streaming verification decoding (SVeD) that triggers captioning when the perplexity of the latest caption on a new frame exceeds a scaled threshold; add peak-end memory compression and a streaming KV cache for long videos.
Key results
- On OmniStar online evaluation, LiveStar achieves SemCor 3.19 for RNG versus 1.68 for VideoLLM-online and 1.93 for MMDuet, with TimDiff 1.91 versus 2.32 for MMDuet; the paper reports an average 19.5% SemCor gain and 18.1% TimDiff reduction.
- On the Ego4D Narration Stream offline setting, LiveStar reaches 61.1% TokAcc versus 52.4% for LION-FS, and PPL 1.97 versus 2.09.
- KV caching yields a 1.53x FPS speedup (3.82 FPS) on 5-minute videos with negligible performance loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LiveStar instead of MMDuet for online narration | Online video assistants · OmniStar-RNG online | SemCor: 1.93 -> 3.19 | Helped |
| Use LiveStar instead of MMDuet for online narration | Online video assistants · OmniStar-RNG online | TimDiff: 2.32 -> 1.91 | Helped |
| Use LiveStar instead of the strongest listed baseline | Online video assistants · OmniStar-FDQ | SemCor: 4.78 -> 6.44 | Helped |
| Fine-tune LiveStar on the SVBench training set | LiveStar · SVBench dialogue and streaming AVG | AVG: 49.76 -> 57.41 | Helped |
| Replace Peak-End memory dropout with Uniform dropout (ablation) | LiveStar · OmniStar-RNG | SemCor: 3.19 -> 3.04 | Hurt |
| Replace Peak-End memory dropout with FIFO dropout (ablation) | LiveStar · OmniStar-RNG | TimDiff: 1.91 -> 2.09 | Hurt |
| Add both intra-dialogue and inter-dialogue KV caches (ablation) | LiveStar with Peak-End memory dropout · OmniStar-RNG | FPS: 2.50 -> 3.82 | Helped |
| Add the inter-dialogue KV cache (ablation) | LiveStar with Peak-End memory dropout · OmniStar-RNG | FPS: 2.92 -> 3.82 | Helped |
| Increase the paraphrased caption pool from M=1 to M=3 (ablation) | LiveStar · OmniStar-RNG | SemCor: 3.19 -> 3.24 | Helped |
| Increase the paraphrased caption pool from M=1 to M=3 (ablation) | LiveStar · OmniStar-RNG | TimDiff: 1.91 -> 1.97 | Hurt |
Surprising findings
- Uniform and FIFO memory dropout both reduce SemCor, with FIFO increasing TimDiff by 9.42%.
- Increasing the caption paraphrase pool size improves SemCor but degrades temporal alignment (TimDiff), so the default is M=1.
Limitations
- Each frame is compressed to 16 visual tokens, which may lose fine-grained visual detail and subtle motion cues.
- Only vision-text modalities are supported; no audio.
- No error bars or statistical significance reported.
Experimental setup
Datasets: ActivityNet Captions Shot2Story Ego4D Narration Stream MVBench OmniStar SVBench
Benchmarks: OmniStar Ego4D Narration Stream SVBench
Metrics: SemCor SumFluen TimDiff TimRedun TimCover PPL TokAcc TimeDiff Fluency FPS
Baselines: VideoLLM-online VideoLLM-MoD MMDuet LION-FS GPT-4o GPT-4V Qwen2.5-VL InternVL2.5 MiniCPM-V 2.6 LLaVA-NeXT-Video
Builds on: VideoLLM-online InternVideo2.5 LLaVA-1.5
Closest papers by meaning
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant (NeurIPS 2025 · similarity 0.90)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.89)
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.89)
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.89)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.