Research map / Video / Video-language understanding
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of video-LMM design choices (sampling, encoders, resampling, training schedule, data mix) that yields Scaling Consistency, the ApolloBench subset, and the Apollo 1.5B/3B/7B video LMM family.
Problem
Video large multimodal models are developed with many unjustified design decisions, and their training and evaluation are computationally expensive, limiting research progress.
Approach
Shows that design decisions validated on smaller LMMs and smaller datasets transfer to larger ones (Scaling Consistency), uses this to run a broad video-LMM design sweep, curates a compact ApolloBench that is ~41x faster to evaluate, and applies the findings to train the Apollo family.
Key results
- Apollo-7B reaches 70.9 on MLVU and 61.2 on Video-MME (w/o subs) among 7B models, per the introduction.
- Apollo-3B scores 58.4 on Video-MME and 68.7 on MLVU, outperforming most existing 7B models.
- ApolloBench evaluation is reported as 41x faster than the full benchmark suite while being highly correlated with existing benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fps sampling instead of uniform frame sampling during training (ablation) | video-LMM · video understanding | overall performance | Helped |
| Use 16–32 tokens per frame while balancing fps and tokens per second (ablation) | video-LMM · video understanding | overall performance: 16-32 tokens per frame | Mixed |
| Use SigLIP-SO400M as the single vision encoder (ablation) | video-LMM · video understanding | overall performance | Helped |
| Combine InternVideo2 with SigLIP-SO400M (ablation) | video-LMM · video understanding | ApolloBench accuracy: ∼7% improvement | Helped |
| Use a Perceiver Resampler instead of 2D conv + average pooling or MLP + average pooling (ablation) | video-LMM · ApolloBench | overall accuracy: 44.7 and 53.2 -> 57.2 | Helped |
| Add text or learnable tokens between video clip features (ablation) | video-LMM · ApolloBench | overall accuracy: 2−3% improvement | Helped |
| Progressively unfreeze model components across three training stages (ablation) | video-LMM · ApolloBench | overall accuracy | Helped |
| Finetune the video encoders (ablation) | video-LMM · video understanding | overall performance | Helped |
| Include a moderate amount of text data in the training mixture (ablation) | video-LMM · video understanding | overall performance: 10 ∼ 14% text data | Helped |
| Use a slightly video-heavy mix of the remaining modalities (ablation) | video-LMM · video understanding | overall performance | Helped |
Surprising findings
- Video encoders outperform image encoders only on temporal perception, and SigLIP-SO400M is the best single encoder overall.
- Fine-tuning vision encoders when the LLM is unfrozen with mixed data significantly hurt performance, contrary to common practice.
Limitations
- Final Apollo training data excludes non-permissive datasets such as ChatGPT-generated ones, limiting comparability.
- Design sweeps use Qwen2.5-3B and a fixed 750K-sample dataset, and scaling consistency holds only above a critical model size (about 2-4B).
Experimental setup
Datasets: Video-MME MLVU TempCompass LongVideoBench NExT-QA Perception-Test
Benchmarks: TempCompass MLVU Perception-Test Video-MME LongVideoBench ApolloBench
Metrics: accuracy
Baselines: GPT-4o Gemini-1.5-Pro LLaVA-OV-7B InternVL2-8B Oryx-7B Kangaroo-8B Video-XL-7B LongVU-7B LongVA-7B Oryx-34B
Builds on: Cambrian-1 Qwen2.5 Perceiver Resampler LLaVA-OneVision
Closest papers by meaning
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.90)
- LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.88)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.