Research map / Vision-language and multimodal / Multimodal LLMs
NVILA: Efficient Frontier Visual Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NVILA, an open 8B/15B VLM that scales visual resolution and frames then compresses tokens, and applies DeltaLoss data pruning, FP8 training, LoRA-style fine-tuning recipes, and W8A8/W4A16 quantized deployment to cut training, prefill, and decode costs while matching leading VLMs.
Problem
VLMs are expensive to train, fine-tune, and deploy, and prior work has focused mostly on accuracy rather than end-to-end efficiency, especially for high-resolution images and long videos.
Approach
A "scale-then-compress" design: scale spatial and temporal resolution (dynamic S2 tiling, more frames) to raise the accuracy ceiling, then compress visual tokens (spatial-to-channel reshape, temporal averaging), combined with full-lifecycle efficiency optimizations.
Key results
- NVILA reduces training cost by 1.9-5.1x relative to LLaVA-OneVision and cuts prefilling latency 1.6-2.2x and decoding latency 1.2-2.8x versus Qwen2-VL on a single RTX 4090.
- NVILA-8B reaches 64.2 on Video-MME (w/o subtitles) with 256 frames, state-of-the-art among open 7-8B models and matching GPT-4o mini.
- 3x3 spatial-to-channel compression with visual encoder pre-training recovers most accuracy lost from compression (DocVQA 82.3 to 88.8) while keeping a 2.4x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale spatial resolution with dynamic tiling (ablation) | NVILA image encoder · DocVQA | score: 40.6 -> 91.1 | Helped |
| add 3×3 spatial token compression (ablation) | NVILA with spatial scaling · DocVQA | score: 91.1 -> 82.3 | Hurt |
| add visual encoder pre-training after 3×3 compression (ablation) | NVILA scale-and-compress model · DocVQA | score: 82.3 -> 88.8 | Helped |
| replace spatial-to-channel compression with TokenLearner (ablation) | NVILA spatial token compression · DocVQA | score: 88.8 -> 86.5 | Hurt |
| replace spatial-to-channel compression with Perceiver Resampler (ablation) | NVILA spatial token compression · DocVQA | score: 88.8 -> 71.8 | Hurt |
| increase sampled frames from 8 to 32 (ablation) | NVILA video model · Video-MME w/o subtitles, Overall | accuracy: 55.7 -> 61.0 | Helped |
| add 4× temporal averaging to the 32-frame model (ablation) | NVILA video model · Video-MME w/o subtitles, Overall | accuracy: 61.0 -> 60.1 | Hurt |
| scale to 256 frames with 8× temporal averaging (ablation) | NVILA video model · Video-MME w/o subtitles, Overall | accuracy: 55.7 -> 64.0 | Helped |
| prune SFT data to 50% with DeltaLoss (ablation) | NVILA recipe · DocVQA | score: 90.1 -> 89.7 | No clear effect |
| replace DeltaLoss with random pruning at 50% keep (ablation) | NVILA recipe · DocVQA | score: 89.7 -> 87.1 | Hurt |
| use FP8 instead of BF16 with gradient checkpointing disabled (ablation) | NVILA training · Training throughput | throughput: 199.2 (1.0×) -> 390.1 (2.0×) | Helped |
| use FP8 instead of BF16 with gradient checkpointing disabled (ablation) | NVILA training · MMMU | score: 47.9 -> 47.0 | Hurt |
Surprising findings
- Naively adding Pixmo data to the NVILA training set degraded DocVQA and TextVQA, showing indiscriminate dataset growth can hurt performance.
- Learnable token compression (TokenLearner, Perceiver Resampler) did not outperform the simple spatial-to-channel reshape at the same token-reduction ratio.
Limitations
- Learnable token compression methods underperform simple reshape, with the cause left as optimization difficulty not yet investigated.
- Many training data sources are mixtures of public datasets; the DeltaLoss pruning depends on reference large and small VLMs.
- Medical and robotics results rely on other frameworks (NVILA-M3, NaVILA) and are reported only briefly.
Experimental setup
Datasets: LLaVA-CC3M-Pretrain ALLaVA Docmatix PDFA LSVT ArT COYO ShareGPT4v MMC4 ShareGPT4V-SFT Molmo The Cauldron
Benchmarks: AI2D ChartQA DocVQA InfoVQA MathVista MMMU RealWorldQA SEED-Bench TextVQA VQAv2 ActivityNet-QA LongVideoBench
Metrics: accuracy TTFT prefilling latency decoding throughput training throughput mean IoU success rate SPL BLEU-4 ROUGE
Baselines: Qwen2-VL InternVL2 LLaVA-OneVision VILA-1.5 Cambrian-1 Florence-VL Pixtral GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet LongVILA Oryx-1.5
Builds on: VILA S2 (scaling on scales) InternVL dynamic resolution AWQ COAT
Closest papers by meaning
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.90)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts (AAAI 2025 · similarity 0.88)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.88)
- CogVLM: Visual Expert for Pretrained Language Models (NeurIPS 2024 · similarity 0.87)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.