Research map / Vision-language and multimodal / Multimodal LLMs
A Comprehensive Overhaul of Multimodal Assistant with Small Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies design choices for multimodal small language models (SLMs) and uses the findings to build Mipha (1.6B–3B), where Mipha-3B matches or beats several 7B and 13B MLLMs without extra training data.
Problem
Multimodal LLMs are expensive to train and run; it is unclear whether design insights from large MLLMs transfer to small language models (under ~3B parameters) in multimodal assistants.
Approach
Empirically dissects the MSLM design space (SLM choice, base vs. instruction-tuned LM, visual backbone and resolution, frozen vs. finetuned components, full tuning vs. LoRA) under a fixed data budget, finds several conclusions that contradict MLLM findings, and combines them into the Mipha model family built on LLaVA-1.5's recipe.
Key results
- Mipha-3B reaches 81.3 on VQAv2 and 69.7 on MMBench (en), exceeding LLaVA-1.5-13B on VQAv2 (80.0) and MMBench (67.7) with the same 665K instruction data.
- Mipha-3B achieves 86.7 on POPE, 0.8 points above LLaVA-1.5-13B (85.9).
- Mipha-3B outperforms Bunny-3B on VQAv2 and GQA while using about 45% of Bunny's training data, per the paper's comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mipha-3B instead of LLaVA-1.5-13B | Mipha-3B · VQAv2 | score: 1.3% | Helped |
| Use Mipha-3B instead of LLaVA-1.5-13B | Mipha-3B · GQA | score: 0.6% | Helped |
| Use Mipha-3B instead of LLaVA-1.5-13B | Mipha-3B · MMBench | score: 2% | Helped |
| Use Mipha-3B instead of LLaVA-1.5-13B | Mipha-3B · MMBench-CN | score: 0.7% | Helped |
| Use Mipha-3B instead of LLaVA-1.5-13B | Mipha-3B · POPE | score: 0.8% | Helped |
| Use SigLIP instead of CLIP as the visual backbone (ablation) | MSLMs · Multimodal benchmarks | performance | Helped |
| Increase image resolution from 384px to 448px (ablation) | SigLIP MSLMs · Five of eight multimodal benchmarks | performance | Hurt |
| Increase image resolution from 384px to 448px (ablation) | SigLIP MSLMs · MM-Vet and seven other benchmarks | performance | Mixed |
| Fine-tune the visual backbone instead of freezing it (ablation) | MSLMs · Evaluated benchmarks | performance | Helped |
| Fine-tune the language model instead of freezing it (ablation) | MSLMs · Various benchmarks | performance | Helped |
| Fine-tune both the visual backbone and language model (ablation) | MSLMs · Certain benchmarks | performance | Helped |
| Use LoRA instead of full-parameter tuning (ablation) | MSLMs · Evaluated benchmarks | performance | No clear effect |
Surprising findings
- Higher image resolution is not always better: a 224px CLIP outperformed 336/448px variants on some benchmarks, and SigLIP 384px beat 448px on five of eight benchmarks.
- Instruction tuning (SFT, RLHF, Safe-RLHF) of the small language model shows no clear-cut benefit over the base LM, contrary to prior MLLM findings.
- Fine-tuning the vision backbone improves MSLM performance, contrary to earlier MLLM findings that it hurts.
Limitations
- Findings may not transfer to newer small language models beyond those studied (Phi, StableLM, Pythia, Gemma).
- Weak on OCR and mathematical problem solving, per the authors.
- Potential for unreliable responses, societal bias and harmful content generation.
Experimental setup
Datasets: LCS-558K LLaVA-1.5 665K instruction mixture Alpaca PKU-SafeRLHF-10K ImageNet21K WebLI LVD-142M
Benchmarks: VQAv2 GQA ScienceQA-IMG TextVQA POPE MME MMBench MMBench-CN MM-Vet SEED-Bench VizWiz MMLU
Metrics: accuracy MME perception score MME cognition score MM-Vet score
Baselines: LLaVA-1.5-7B LLaVA-1.5-13B Qwen-VL MobileVLM-v2-3B Bunny-3B MoE-LLaVA-3.6B InstructBLIP mPLUG-Owl2
Builds on: LLaVA-1.5 Phi-2 SigLIP LoRA Safe-RLHF
Closest papers by meaning
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.