Research map / Vision-language and multimodal / Multimodal LLMs
Scaling Laws for Native Multimodal Models
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, Alaaeldin El-Nouby
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use early fusion instead of late fusion | native multimodal model 1.5B · SFT LLaVA mixture | AVG accuracy: 46.8 -> 47.6 | Improved |
| add mixture of experts (8 experts, top-1) to early fusion (ablation) | early-fusion NMM 1.5B · SFT LLaVA mixture | AVG accuracy: 47.6 -> 48.2 | Improved |
| add MoEs to early fusion (ablation) | early-fusion NMM 1.5B · SFT LLaVA mixture, COCO | CIDEr: 72.0 -> 69.6 | Hurt |
| add MoEs to early fusion (ablation) | early-fusion NMM 1.5B · SFT LLaVA mixture, TextVQA | accuracy: 28.1 -> 30.0 | Improved |
| freeze vision encoder instead of lr scaler 1 when initialized from pretrained (ablation) | late-fusion NMM · SFT average | AVG (SFT) accuracy: 43.49 -> 54.14 | Improved |
| lower vision encoder lr scaler from 0.1 to 0.01 when training from scratch (ablation) | late-fusion NMM · SFT average | AVG (SFT) accuracy: 34.84 -> 38.77 | Improved |
| use sparse MoE instead of dense for loss-compute fit | early-fusion NMM · pretraining validation loss | multiplicative constant: 29.574 -> 27.086 | Improved |
| use late fusion instead of early fusion for loss scaling exponent | NMM · pretraining validation loss | exponent c: -0.0492 -> -0.0494 | No clear effect |
| use sparsity (MoE) for loss scaling exponent | early-fusion NMM · pretraining validation loss | exponent c: -0.049 -> -0.048 | No clear effect |
| use modality-agnostic MoE routing instead of modality-aware routing (ablation) | sparse early-fusion NMM · image-caption and interleaved validation loss | validation loss | Improved |
| raise image-caption proportion in training mixture (30 to 40 percent) (ablation) | early-fusion NMM · scaling exponent for image-caption | absolute exponent: 0.056 -> 0.061 | Improved |
| increase image resolution (ablation) | early vs late fusion · pretraining loss | validation loss | Mixed |
Closest papers by meaning
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.90)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.90)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.89)
- Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (ICLR 2025 · similarity 0.88)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
