Research map / Vision-language and multimodal / Multimodal LLMs
"MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training"
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Samuel Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Futang Peng, Anton Belyi, Max A Schwarzer, Hongyu Hè, Xianzhi Du and 17 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase image resolution from 224 to 336 (ablation) | MM1 image encoder · pre-training few-shot avg | score: approx. 3% boost in all metrics | Improved |
| increase image encoder from ViT-L to ViT-H (ablation) | MM1 image encoder · pre-training few-shot avg | score: usually less than 1% | Improved |
| add VeCap-300M synthetic captions to encoder training (ablation) | MM1 image encoder · pre-training few-shot | score: more than 1% in few-shot | Improved |
| use contrastive CLIP encoder instead of reconstructive AIM (ablation) | MM1 image encoder · pre-training few-shot | score: 0.3% to 1.5% | Improved |
| increase number of image tokens (64 to 144) or resolution (ablation) | VL connector · 0/4/8-shot | score | Improved |
| change VL connector type (avg pooling, attention pooling, C-Abstractor) (ablation) | MM1 · 0/4/8-shot and SFT | score | No clear effect |
| include at least 50% interleaved data in pre-training (ablation) | MM1 pre-training mix · 4/8-shot | accuracy: 8-shot 45% -> over 61%; 4-shot 43.7% -> 58% | Improved |
| increase caption data share in pre-training (ablation) | MM1 pre-training mix · 0-shot | accuracy: 25.8% -> 39.3% | Improved |
| add text-only data to captioned data (ablation) | MM1 pre-training mix · few-shot and TextCore | score | Improved |
| add text-only data to interleaved data (ablation) | MM1 pre-training mix · few-shot | score: minor drop | Hurt |
| add VeCap synthetic captions to caption data (ablation) | MM1 pre-training mix · few-shot | score: 2.4% and 4% absolute | Improved |
| raise SFT resolution from 336 to 1344 (ablation) | MM1 SFT · SFT meta-average | average score: 15% relative increase | Improved |
Closest papers by meaning
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.94)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.92)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.92)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.92)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
