Research map / Video / Video-language understanding
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LION-FS, a fast-slow online video assistant that routes dual-encoder features (SigLIP + EgoVLPv2) for per-frame response determination and adds training-free multi-granularity keyframe augmentation for precise response generation on Ego4D and Ego-Exo4D.
Problem
Online video assistants trade off response accuracy against real-time efficiency: low-frame-rate, coarse image tokens hurt when to respond and what to say in first-person video streams.
Approach
Decouple response determination (Fast Path: token aggregation router fusing third-person SigLIP and first-person EgoVLPv2 tokens at higher frame rate, plus token dropping router) from response generation (Slow Path: training-free grid and hand-object box token augmentation on keyframes inserted into a thinking template).
Key results
- On Ego4D Narration Validation, LION-FS reaches LM-Correctness 52.4% vs 49.0% for VideoLLM-online and LL-PPL 2.09 vs 2.40.
- On Ego-Exo4D Narration Validation, LION-FS improves Fluency to 36.5% and LM-Correctness to 48.2% over VideoLLM-online (33.7%, 44.8%).
- LION-FS processes video streams at four times the frame rate of prior methods, with dropping at 0.5 ratio giving a 1.12x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace either single-encoder representation with adaptive aggregation of SigLIP and EgoVLPv2 tokens (ablation) | Fast Path visual encoding · Ego-Exo4D Narration Validation | Fluency: 33.7% -> 38.1% | Helped |
| concatenate the 10-token SigLIP and 10-token EgoVLPv2 features (ablation) | Fast Path visual encoding · Ego-Exo4D Narration Validation | TimeDiff: 0.78 -> 1.65 | Hurt |
| add the CLS token from one encoder to the CLS token from the other encoder (ablation) | Fast Path visual encoding · Ego-Exo4D Narration Validation | LL-PPL: 2.25 -> 2.18; 2.25 -> 2.38 | Mixed |
| use EgoVLPv2 or both encoders as Visual Guidance instead of SigLIP CLS (ablation) | Token Aggregation Router · Ego-Exo4D Narration Validation | TimeDiff: 0.64 -> 0.70; 0.64 -> 1.08 | Hurt |
| remove the Token Dropping Router (ablation) | LION-FS Fast Path · Ego-Exo4D Narration Validation | LL-PPL: 2.16 -> 2.25 | No clear effect |
| replace learned token dropping with random dropping (ablation) | LION-FS Fast Path · Ego-Exo4D Narration Validation | LL-PPL: 2.25 -> 15.48 | Hurt |
| drop tokens in all layers rather than interleaved layers (ablation) | Token Dropping Router · Ego-Exo4D Narration Validation | Fluency: 36.5% -> 34.1% | Hurt |
| drop tokens in deep layers rather than interleaved layers (ablation) | Token Dropping Router · Ego-Exo4D Narration Validation | Fluency: 36.5% -> 35.6% | No clear effect |
| drop tokens in interleaved and deep layers (ablation) | Token Dropping Router · Ego-Exo4D Narration Validation | Fluency: 36.5% -> 33.9% | Hurt |
| increase token dropout rate from β = 0.2 to β = 0.5 (ablation) | Token Dropping Router · Ego-Exo4D Narration Validation | LL-PPL: 2.21 -> 2.16 | Helped |
| increase token dropout rate from β = 0.5 to β = 0.8 (ablation) | Token Dropping Router · Ego-Exo4D Narration Validation | LL-PPL: 2.16 -> 2.28 | Hurt |
| add fine-grained 1×6×6 tokens to keyframes (ablation) | LION-FS Slow Path · Ego-Exo4D Narration Validation | LL-PPL: 2.16 -> 2.07 | Helped |
Surprising findings
- Simply concatenating the two encoders' tokens hurt TimeDiff and did not consistently improve results, motivating adaptive routing instead.
Limitations
- Response determination still slightly underperforms baseline on TimeDiff for Ego4D, attributed to short average response length.
- Relies on training-free slow-path augmentation whose benefit depends on the detector quality for hands and objects.
Experimental setup
Datasets: Ego4D narration stream Ego-Exo4D narration
Benchmarks: Ego4D Narration Validation Ego-Exo4D Narration Validation
Metrics: LL-PPL TimeDiff Fluency LM-Correctness
Baselines: VideoLLM-online VideoLLM-MoD
Builds on: VideoLLM-online Mixture-of-Depths SlowFast-LLaVA Thinking, Fast and Slow
Closest papers by meaning
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.87)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.86)
- Online Video Understanding: OVBench and VideoChat-Online (CVPR 2025 · similarity 0.85)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.85)
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.85)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.