AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

"MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training"

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Samuel Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Futang Peng, Anton Belyi, Max A Schwarzer, Hongyu Hè, Xianzhi Du and 17 more

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase image resolution from 224 to 336 (ablation)MM1 image encoder · pre-training few-shot avgscore: approx. 3% boost in all metricsImproved
increase image encoder from ViT-L to ViT-H (ablation)MM1 image encoder · pre-training few-shot avgscore: usually less than 1%Improved
add VeCap-300M synthetic captions to encoder training (ablation)MM1 image encoder · pre-training few-shotscore: more than 1% in few-shotImproved
use contrastive CLIP encoder instead of reconstructive AIM (ablation)MM1 image encoder · pre-training few-shotscore: 0.3% to 1.5%Improved
increase number of image tokens (64 to 144) or resolution (ablation)VL connector · 0/4/8-shotscoreImproved
change VL connector type (avg pooling, attention pooling, C-Abstractor) (ablation)MM1 · 0/4/8-shot and SFTscoreNo clear effect
include at least 50% interleaved data in pre-training (ablation)MM1 pre-training mix · 4/8-shotaccuracy: 8-shot 45% -> over 61%; 4-shot 43.7% -> 58%Improved
increase caption data share in pre-training (ablation)MM1 pre-training mix · 0-shotaccuracy: 25.8% -> 39.3%Improved
add text-only data to captioned data (ablation)MM1 pre-training mix · few-shot and TextCorescoreImproved
add text-only data to interleaved data (ablation)MM1 pre-training mix · few-shotscore: minor dropHurt
add VeCap synthetic captions to caption data (ablation)MM1 pre-training mix · few-shotscore: 2.4% and 4% absoluteImproved
raise SFT resolution from 336 to 1344 (ablation)MM1 SFT · SFT meta-averageaverage score: 15% relative increaseImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy