Research map / Architectures and efficiency / Attention and transformers
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoLLM-MoD, which applies mixture-of-depths routing to vision tokens in a streaming video LLM so most vision tokens skip deep layers, cutting training FLOPs and memory while matching or beating full-computation training.
Problem
Scaling vision tokens per frame for online/streaming video LLMs makes training and inference cost grow quickly with video length, while learnable token-reduction methods (Q-Former, Perceiver Resampler) ignore the causal LLM context (KV cache).
Approach
Instead of reducing the number of vision tokens, a per-layer LayerExpert linear router scores each frame's vision tokens and only the top-k (keep ratio r=0.2) are processed by attention and FFN in alternating layers; the rest bypass the layer through the residual connection, so the full context is kept while compute drops.
Key results
- On Ego4D narration stream, the method reaches LM-PPL 2.41, TimeDiff 2.04, Fluency 45.2%, LM-Correctness 48.9% with 30.74T FLOPs versus 48.29T for full computation (about 0.6x FLOPs, 14 vs 24 hrs training).
- On COIN, the method attains step recognition 63.4 and task 92.8, the best among end-to-end models shown, and on EgoExo4D keystep it reaches 44.85/42.62 accuracy vs 40.53/40.73 for VideoLLM-online.
- The abstract reports about 42% training time and about 30% memory savings, and 1.7x longer supported video at inference via KV cache savings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learnable LayerExpert selection at keep ratio r=0.2 instead of full computation | VideoLLM-MoD · Ego4D Narration Stream Validation | FLOPs: 48.29T -> 30.74T | Helped |
| Use learnable LayerExpert selection at keep ratio r=0.2 instead of full computation | VideoLLM-MoD · Ego4D Narration Stream Validation | Fluency: 45.3% -> 45.2% | No clear effect |
| Use learnable LayerExpert selection at keep ratio r=0.2 instead of full computation | VideoLLM-MoD · COIN Step recognition | Top-1 accuracy: 63.1 -> 63.4 | Helped |
| Use learnable LayerExpert selection at keep ratio r=0.2 instead of full computation | VideoLLM-MoD · COIN Next forecasting | Top-1 accuracy: 49.1 -> 49.7 | Helped |
| Insert LayerExpert in every layer instead of every other layer (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | Fluency: 45.2% -> 42.5% | Hurt |
| Insert LayerExpert in every layer instead of every other layer (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | LM-Correctness: 48.9% -> 47.5% | Hurt |
| Restrict LayerExpert to deep layers instead of interleaving it across layers (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | Fluency: 45.2% -> 45.2% | No clear effect |
| Select vision tokens using learned importance scores rather than randomly (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | TimeDiff: 2.18 -> 2.04 | Helped |
| Select vision tokens using learned importance scores rather than uniformly (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | TimeDiff: 2.17 -> 2.04 | Helped |
| Reduce vision keep ratio from r=0.2 to r=0.1 (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | TimeDiff: 2.04 -> 2.11 | Hurt |
| Increase vision keep ratio from r=0.2 to r=0.3 (ablation) | VideoLLM-MoD · Ego4D Narration Stream Validation | TimeDiff: 2.04 -> 2.05 | No clear effect |
Surprising findings
- Skipping ~80% of vision tokens in layers slightly improves performance over full-computation training, which the authors attribute to a harder routing objective reducing overfitting.
- Early exit at layer 2 and LayerSkip-style skipping of all vision tokens in alternate layers degrade performance substantially, showing selection matters.
Limitations
- Experiments focus on egocentric and instructional videos; exocentric datasets were not extensively evaluated.
- Gains over the full-computation baseline are small on several metrics and some COIN metrics (proc.+) are lower than the baseline.
Experimental setup
Datasets: Ego4D EgoExo4D COIN LiveChat
Benchmarks: Ego4D Narration Stream Ego4D LTA EgoExo4D Fine-grained Keystep Recognition COIN
Metrics: LM-PPL TimeDiff Fluency LM-Correctness ED@Z=20 Top-1 accuracy FLOPs training time GPU memory
Baselines: VideoLLM-online Full-computation EarlyExit LayerSkip TimeSformer VideoTF Palm AntGPT
Builds on: mixture-of-depths VideoLLM-online LayerSkip LLaVA LoRA
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.88)
- VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.