Research map / Video / Video-language understanding
Exploring the Design Space of Visual Context Representation in Video MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies how to allocate a fixed visual context budget between number of frames and visual embeddings per frame in video MLLMs, fits scaling-law curves, derives an optimal allocation formula, and releases Opt-Visor (7B, up to 162 frames).
Problem
Visual context representation for video MLLMs (how many frames to select and how many visual tokens per frame to keep) is under-studied, and the two factors compete for a limited LLM context window.
Approach
Formulates visual context allocation as a constrained optimization (T x M < L), fits power-law and linear scaling curves for frames and embeddings under sampling and mean-pooling compression, and uses a joint loss fit with Lagrange multipliers to derive the optimal frame/token split.
Key results
- Opt-Visor-7B (120 frames x 49 tokens) achieves 54.9 on VideoMME w/o subs, 63.6 on MLVU, and 51.63 on LongVideoBench, best among listed open-source models.
- Under a fixed 5880-token budget, 120 frames with 49 embeddings per frame gives the lowest loss (0.639), close to the derived optimum of about 118 frames and 51 tokens at 6K context.
- Compression-based (4x4 mean pooling) with 196 tokens/frame reaches average 47.11 vs 44.53 for best sampling-based setting (49 tokens).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sampled embeddings per frame from 1 to 49 (ablation) | Sampling-based video MLLM · 32-frame visual context; reported benchmark average | Avg. accuracy: 36.73 -> 44.53 | Helped |
| Increase sampled embeddings per frame from 49 to 196 (ablation) | Sampling-based video MLLM · 32-frame visual context; reported benchmark average | Avg. accuracy: 44.53 -> 43.70 | Hurt |
| Use mean pooling rather than sampling to select embeddings (ablation) | Video MLLM · 32 frames, 196 embeddings per frame; reported benchmark average | Avg. accuracy: 43.70 -> 47.11 | Helped |
| Increase mean-pooled embeddings per frame from 1 to 196 (ablation) | Compression-based video MLLM · 32-frame visual context; reported benchmark average | Avg. accuracy: 36.15 -> 47.11 | Helped |
| Increase uniformly sampled frames from 8 to 128 (ablation) | Sampling-based video MLLM · 49 embeddings per frame; reported benchmark average | Avg. accuracy: 37.67 -> 49.33 | Helped |
| Increase mean-pooled frames from 8 to 128 (ablation) | Compression-based video MLLM · 49 embeddings per frame; reported benchmark average | Avg. accuracy: 42.30 -> 49.33 | Helped |
| Allocate a fixed visual-token budget to 120 frames and 49 embeddings per frame instead of 8 frames and 729 embeddings per frame (ablation) | Video MLLM · Same total visual embeddings; reported benchmark average | Avg. accuracy: 38.16 -> 48.97 | Helped |
| Allocate a fixed visual-token budget to 162 frames and 36 embeddings per frame instead of 120 frames and 49 embeddings per frame (ablation) | Video MLLM · Same total visual embeddings; reported benchmark average | Avg. accuracy: 48.97 -> 50.62 | Helped |
| Use the derived optimal allocation of 120 frames and 49 embeddings per frame (ablation) | Video MLLM · Same total visual embeddings; language modeling loss | language modeling loss: 0.653 -> 0.639 | Helped |
| Add pre-training before visual instruction tuning (ablation) | Video MLLM with Qwen2-7B · Video MLLM evaluation; specific benchmark not reported | performance | No clear effect |
| Increase frames beyond 64 for episodic reasoning (ablation) | Video MLLM · Event-Bench | accuracy | No clear effect |
| Replace sampling with mean pooling for temporal frame compression (ablation) | Video MLLM · Same number of frames; video understanding benchmarks | accuracy | Helped |
Surprising findings
- Lower language modeling loss with 196 embeddings per frame did not translate to better downstream accuracy than 49 embeddings with sampling-based selection.
- Performance lost from compressing embeddings per frame can be recovered by adding more frames, e.g., 128 frames x 49 tokens surpasses 32 frames x 196 tokens at equal total token count.
Limitations
- Experiments restricted to the LLaVA-like architecture with Qwen2-7B and SigLIP; other backbones and encoders not tested.
- Only language modeling loss and zero-shot benchmark accuracy used as metrics.
- Next-token loss does not fully track downstream performance, so the derived optimum is a starting point.
Experimental setup
Datasets: Cauldron VideoChatGPT-100K ShareGPT4Video ShareGPTVideo VIM NExT-QA SthSthV2 STAR TextVR CLEVRER Kinetics-710 VideoChat2 instructions
Benchmarks: Event-Bench NBench/VNBench MLVU VideoMME LongVideoBench
Metrics: language modeling loss zero-shot accuracy R^2 of fit mean squared error of fit
Baselines: GPT-4o Gemini-1.5-Pro GPT-4V Qwen-VL-Max Video-CCAM-14B Video-LLaVA-7B LLaMA-VID-long-7B MovieChat-7B VideoChat2-7B ST-LLM-7B VideoLLaMA2-7B LongVA-7B
Builds on: LLaVA Chinchilla scaling law LLaVA-OneVision AnyRes
Closest papers by meaning
- Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.90)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.88)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.