Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Shengbang Tong, David Fan, John Nguyen, Ellis L Brown II, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis and 8 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a global shared expert (ablation) | MoE 256 experts, 16 active · Notes | perplexity: 27.392 -> 27.249 | Improved |
| use per-modality shared experts instead of a global one (ablation) | MoE 256 experts · Notes | perplexity: 27.249 -> 27.161 | Improved |
| use per-modality shared experts instead of none (ablation) | MoE 256 experts · GenEval | score: 0.360 -> 0.367 | Improved |
| add loss centering | SigLIP 2 unified model · DPGBench | DPG score: 0.570 -> 0.612 | Improved |
| add loss centering | SigLIP 2 unified model · DCLM val | perplexity: 15.06 -> 15.36 | Hurt |
| add loss centering | SD-VAE unified model · DPGBench | DPG score: 0.470 -> 0.480 | Improved |
| replace shared FFN with modality-specific FFN (ablation) | Transfusion baseline · DCLM val | perplexity: 15.93 -> 15.13 | Improved |
| replace shared FFN with modality-specific FFN (ablation) | Transfusion baseline · DPGBench | DPG score: 0.45 -> 0.47 | Improved |
| use SigLIP 2 (RAE) instead of VAE encoders (ablation) | modality-specific FFN model · DPGBench | DPG score: 0.47 -> 0.57 | Improved |
| use MoE for parameter separation (vs dense, MoT) (ablation) | SigLIP 2 model · DPGBench | DPG score: 0.63 | Improved |
| switch from v-pred to x-pred (ablation) | MoE + SigLIP 2 · DPGBench | DPG score: 0.63 -> 0.65 | Improved |
| train jointly with language (multimodal vs T2I-only) | dense model · DPGBench | DPG score: 0.598 -> 0.622 | Improved |
Closest papers by meaning
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.92)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.91)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.91)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.91)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
