Research map / Video / Video-language understanding
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LongVILA, a full-stack long-context VLM solution combining a five-stage training pipeline (including long-context LLM extension and long-video SFT) with the MM-SP sequence-parallel training/inference system, scaling VILA to 2048 frames and 2M-token training.
Problem
Long-context vision-language models for long video understanding are hard to train: long video sequences are memory- and compute-intensive, existing sequence-parallel systems handle multimodal inputs and heterogeneous networks poorly, and there is little long-video instruction data.
Approach
Co-design of a five-stage training curriculum (LLM context extension to 262K tokens plus a long-video SFT dataset) and a multi-modal sequence parallelism system that uses two-stage sharding (balanced image encoding, then token-level sharding) and 2D-attention (intra-node All-to-All plus inter-node P2P) to scale context beyond attention-head limits.
Key results
- LongVILA-7B achieves 65.1% on VideoMME with subtitles and reports strong results on 9 video benchmarks, e.g., 67.7 on EgoSchema and 80.7 on NExT-QA.
- The 2048-frame LongVILA reaches 99.8% accuracy on a 6,000-frame (over 1M token) video needle-in-a-haystack test.
- MM-SP achieves 2.1x-5.7x training speedup over ZigZag ring attention and 1.1x-1.4x over Megatron hybrid CP+TP, and supports over 2M context on 256 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train LongVILA-7B with 256 video frames (ablation) | LongVILA-7B · VideoMME long videos, with subtitles | accuracy | Helped |
| Increase frames from 8 to 256 (ablation) | LongVILA · LongVILA-Caption | average score: 2.00 -> 3.26 | Helped |
| Use two-stage rather than one-stage sharding (ablation) | MM-SP training · Video captioning dataset | iteration time: 0.78 -> 0.77; 0.89 -> 0.86; 1.20 -> 1.12 | Helped |
| Enable communication overlap in Ring-style sequence parallelism (ablation) | Ring-style attention kernel · 4K–32K sequence lengths | forward and backward attention kernel wall-clock time: forw. w/o 29.5 -> forw. w/ 35.0 (+18.6%); backw. w/o 77.7 -> backw. w/ 82.2 (+5.8%) | Hurt |
| Use MM-SP rather than Ring-style sequence parallelism | Long-context VLM training · 32 H100 GPUs | throughput: 2.1\times to 5.7\times | Helped |
| Use MM-SP rather than Megatron-LM hybrid context and tensor parallelism | Long-context VLM training · 32 H100 GPUs | throughput: 1.1\times to 1.4\times | Helped |
| Use MM-SP rather than HuggingFace Pipeline parallelism | Long-context VLM inference · 8 H100 GPUs | speedup: 8.2\times | Helped |
| Train with MM-SP rather than pure data parallelism (ablation) | LongVILA · Shot2Story training loss | training loss | No clear effect |
Surprising findings
- The communication overlap design in Ring-style sequence parallelism slows down the attention kernel by occupying SM resources (Table on kernel timing).
- Vanilla data parallelism and DeepSpeed-Ulysses scaling is limited by attention head count, so the 8B model supports far shorter context than the proposed approach at 256 GPUs.
Limitations
- Training system implemented in Triton, with potential further speedups from a C++ port (as stated by the authors).
- Megatron-LM baseline does not natively support VLM training, so main system comparisons report only the LLM backbone without the visual encoder.
Experimental setup
Datasets: SlimPajama Shot2Story COYO-25M YouCook2 ShareGPTVideo VideoMME ActivityNet-QA EgoSchema EventBench LongVideoBench PerceptionTest MVBench
Benchmarks: ActivityNet-QA EgoSchema EventBench LongVideoBench PerceptionTest MVBench NExT-QA VNBench VideoMME Needle-in-a-Haystack (video) LongVILA-Caption VQA-v2
Metrics: accuracy throughput (iterations/s) time per iteration max sequence length caption score (0-5)
Baselines: ZigZag-RingAttn DeepSpeed-Ulysses Megatron-LM CP Megatron-LM CP+TP HuggingFace Pipeline LongVA VideoLLaMA2 Kangaroo LLaVA-OV GPT-4o Gemini-1.5-Pro
Builds on: VILA DeepSpeed-Ulysses ring attention LongVA USP LoongTrain
Closest papers by meaning
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.91)
- World Model on Million-Length Video And Language With Blockwise RingAttention (ICLR 2025 · similarity 0.91)
- NVILA: Efficient Frontier Visual Language Models (CVPR 2025 · similarity 0.90)
- Scaling RL to Long Videos (NeurIPS 2025 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.