Research map / Vision-language and multimodal / Cross-modal learning

Scaling Laws for Multimodal Data Mixtures

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a unified, repetition-aware scaling law for text, vision, and audio data mixtures in MoE pretraining, fit on 268 runs (457M–8.3B total params, up to 150B tokens).

Problem

Understanding how to mix text, vision, and audio data during native multimodal MoE pretraining, accounting for cross-modal interference, capacity sharing, and repetition of scarce non-text data.

Approach

Extends the Chinchilla-form loss with three terms: a log-linear capacity-sharing factor on parameters N (capacity dilution), a directional cross-modal transfer factor on effective data, and a exponential-saturating repetition function phi_m(r) on unique tokens, reducing to Chinchilla in the unimodal limit.

Key results

• Audio is model-heavy with compute-optimal split N_opt ~ C^0.62 vs D_opt ~ C^0.38, while text and vision are balanced (0.50/0.50).
• Repeating vision or audio data beyond roughly 2 epochs yields diminishing returns; the repetition ceiling is about 2.0 fresh-equivalent epochs for vision and 2.76 for audio.
• Cross-modal transfer is asymmetric: vision interferes strongly with audio (v_vision->audio = -0.795) while audio's transfer to text and vision is not stable across folds; on held-out runs 8 of 9 predictions are within 2.6% of observed loss.

Surprising findings

• Audio's transfer to text and vision is not statistically stable across leave-one-scale-out folds, so audio is treated as inert for the other modalities.
• Vision and audio repetition saturates after about two epochs, unlike the up-to-4x repetition tolerance reported for text.

Limitations

• Validated only up to about 10B active parameters and 1T tokens; no long-form video or other modalities.
• Fitted coefficients depend on tokenizers, loss objective, and MoE hyperparameters; treated as a framework rather than universal law.
• Bidirectional capacity-sharing variant did not consistently improve extrapolation; some coefficients have high variance.

Experimental setup

Datasets: FineWeb-HQ FineWeb2-HQ MegaMath-Web-Pro Stack v2-Edu LLaVA-OneVision-1.5 Mid-Training Unsupervised People's Speech People's Speech WenetSpeech Emilia-YODAS Multilingual LibriSpeech Common Voice 24

Metrics: validation loss RMSE_log maximum relative error Pearson correlation

Baselines: Chinchilla TGL NMM SODA

Builds on: Chinchilla scaling laws Step Law Mistral-Nemo tokenizer Megatron-LM bimodal scaling laws

Closest papers by meaning

• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
• Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
• CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.87)
• Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.87)