Research map / Vision-language and multimodal / Multimodal LLMs
NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies design choices and scaling of end-to-end native MLLMs under a data constraint, and builds NaViL (2B and 9B) with LLM initialization, modality-specific MoE, and a jointly scaled visual encoder.
Problem
Native (end-to-end) multimodal LLMs are usually evaluated assuming abundant resources; it is unclear how to build them to match top compositional MLLMs under practical data and compute budgets, and how vision encoder and LLM size should scale together.
Approach
Systematic ablation of native MLLM design (LLM initialization, modality-specific MoE with attention and FFN experts, encoder depth/width) under a data constraint, finding that the optimal visual encoder size grows log-linearly with LLM size, then training NaViL end-to-end on that basis.
Key results
- NaViL-2B (2.4B activated params) reaches an average of 67.1 on the general MLLM benchmarks, the best among native MLLMs and close to compositional InternVL-2.5-2B (67.0).
- NaViL-9B reaches an average of 77.0 on general MLLM benchmarks versus 77.3 for InternVL-2.5-8B, using 450B training tokens versus more than 3.5T for InternVL2.5-8B.
- Finding: the optimal visual encoder size scales log-linearly with LLM size, and scaling the encoder alone shows diminishing returns bounded by LLM capacity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize the LLM from a pre-trained checkpoint instead of training it from scratch (ablation) | Native MLLM · Multimodal validation and zero-shot image captioning | validation loss and caption performance: over 10x more data | Helped |
| Add modality-specific MoE experts to the LLM (ablation) | Native MLLM with a visual encoder · Multimodal validation | validation loss: 1/10 of the data | Helped |
| Vary visual encoder depth and width at a fixed parameter count (ablation) | Native MLLM · Multimodal validation and zero-shot image captioning | validation loss and caption performance: 600M | Mixed |
| Scale up the LLM while keeping the visual encoder fixed (ablation) | Native MLLM · Multimodal validation | validation loss | Helped |
| Increase visual encoder size while keeping the LLM fixed (ablation) | Native MLLM · Multimodal validation | validation loss: 75M | Mixed |
| Scale the visual encoder jointly with LLM size rather than using one encoder size across LLM scales (ablation) | Native MLLM · Multimodal validation | validation loss | Helped |
Surprising findings
- Scaling the visual encoder alone yields diminishing returns, suggesting that MLLM performance is capped by LLM capacity.
- NaViL-2B outperforms the compositional InternVL-2.5-2B on MMVet and CCBench despite using a smaller encoder that is not distilled from a large teacher.
Limitations
- Scaling study only goes up to 9B parameters, so larger scales are untested.
- Only visual and linguistic modalities are studied.
- NaViL-9B was trained on only 450B tokens due to limited compute, and the paper notes a remaining gap to the best compositional MLLMs on VQA tasks.
Experimental setup
Datasets: LAION-2B COYO-700M Wukong SA-1B InternVL-2.5 high-quality data InternLM2.5 pure language data
Benchmarks: MMVet MMMU MMBench-EN MME MathVista OCRBench CCBench TextVQA ScienceQA-IMG GQA DocVQA AI2D
Metrics: accuracy validation teacher-forcing loss normalized average score
Baselines: InternVL-2.5-2B Qwen2VL-2B Mono-InternVL SAIL EVEv2 EVE-7B Emu3 Chameleon-7B InternVL-2.5-8B Qwen2.5-VL
Builds on: InternLM2 Mono-InternVL InternVL-2.5 Qwen3
Closest papers by meaning
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.90)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.