Research map / Vision-language and multimodal / Multimodal LLMs
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents MM1.5, a family of 1B-30B dense and MoE multimodal LLMs built via a data-centric recipe (OCR continual pre-training, optimized SFT mixtures, dynamic image splitting), plus MM1.5-Video and MM1.5-UI variants.
Problem
Building strong multimodal LLMs, especially small ones, requires understanding which data mixtures and training stages drive text-rich image understanding, referring/grounding, and multi-image reasoning; the impact of these choices is under-explored.
Approach
Keep the MM1 architecture but systematically ablate data mixtures across the full lifecycle: an added high-resolution OCR continual pre-training stage, category-balanced SFT mixtures with refer&ground data, adjusted pre-training ratios (50:10:40), and dynamic image splitting for up to 4MP inputs.
Key results
- MM1.5-30B improves DocVQA from 75.8 (MM1) to 91.4 and InfoVQA from 47.3 to 67.3, and MuirBench from 36.7 to 58.2.
- MM1.5-3B-MoE surpasses MM1.5-7B on knowledge, general, referring/grounding and multi-image benchmarks, per the paper.
- MM1.5-3B reaches DocVQA 87.7 and InfoVQA 58.5 versus Phi-3-Vision 83.3 and 49.0, and on VL-ICL averages 56.3 vs Phi-3-Vision's 19.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add high-quality text-rich SFT data (ablation) | MM1.5-3B · Text-rich and knowledge benchmarks | category average score | Helped |
| add science SFT data (ablation) | MM1.5-3B · Knowledge benchmarks | category average score | Helped |
| add refer&ground SFT data (ablation) | MM1.5-3B · Referring and grounding benchmarks | category average score | Helped |
| add refer&ground SFT data (ablation) | MM1.5-3B · General, text-rich, and knowledge benchmarks | category average score | Hurt |
| increase the multi-image SFT sampling ratio (ablation) | MM1.5-3B · Multi-image and MMBase | category average score | Mixed |
| use text-only SFT sampling weight 0.1 (ablation) | MM1.5-3B · Base capabilities | MMBase score | No clear effect |
| reduce continual pre-training image resolution from 1344×1344 (ablation) | MM1.5-3B · Final downstream benchmarks | average score | Hurt |
| add public synthetic captions from ShareGPT4V-PT or LLaVA-Recap-3M to OCR continual pre-training (ablation) | MM1.5-3B · MMBase | MMBase score | No clear effect |
| replace the prior text-only pre-training data with HQ-Text (ablation) | MM1.5 · Knowledge benchmarks | knowledge average score: +0.85 point | Helped |
| change pre-training ratio from 45:45:10 to 50:10:40 (ablation) | MM1.5 · Text-rich benchmarks | text-rich average score: +0.85 | Helped |
| change pre-training ratio from 45:45:10 to 50:10:40 (ablation) | MM1.5 · Knowledge benchmarks | knowledge average score: +0.99 | Helped |
| change pre-training ratio from 45:45:10 to 50:10:40 (ablation) | MM1.5 · Multi-image benchmarks | multi-image average score: -0.05 | Hurt |
Surprising findings
- Continual pre-training at 378x378 resolution can underperform having no continual pre-training at all.
- Adding high-quality synthetic captions (ShareGPT4V-PT, LLaVA-Recap-3M) on top of OCR data gave no conclusive improvement over OCR data alone.
Limitations
- Ablations use the 3B dense model only, with reduced settings for faster iteration.
- Several competitor results were re-run by the authors with custom inference settings, and some baselines could not be evaluated on all benchmarks (e.g., MiniCPM-V2 multi-image).
- Synthetic caption benefits were inconclusive in the initial study and the in-house captioning study was deferred.
Experimental setup
Datasets: PDFA IDL Rendered-text DocStruct-4M ShareGPT4V-PT LLaVA-Recap-3M HQ-Text DocVQA InfoVQA ChartQA GRIT ShareGPTVideo
Benchmarks: MME SEED-Bench POPE LLaVA-Bench-in-the-Wild MM-Vet RealWorldQA WTQ TabFact OCRBench ChartQA TextVQA DocVQA
Metrics: accuracy CIDEr GPT-assisted score Recall@1 (IoU>0.5) ANLS F1 Category Average Score MMBase score
Baselines: MiniCPM-V 2.0 Phi-3-Vision InternVL2 LLaVA-OneVision Cambrian-34B LLaVA-NeXT-34B Ferret-13B Ferret-V2-13B GPT-4o GPT-4V Gemini-1.5-Pro MM1
Builds on: MM1 LLaVA any-resolution (AnyRes) GShard C-Abstractor
Closest papers by meaning
- A Comprehensive Overhaul of Multimodal Assistant with Small Language Models (AAAI 2025 · similarity 0.91)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.