Research map / Large language models / Language, safety and interpretability
LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a 7B multimodal LLM into sparse-MoE small MLLMs (LLaVA-MoD, 1.8B/0.5B Qwen students) via two-stage mimic KD and KTO-based preference distillation.
Problem
Training small-scale multimodal LLMs that are both efficient and strong, since small MLLMs are limited by model capacity and prior work relied mainly on data curation rather than teacher knowledge transfer.
Approach
Upcycle the student LLM's FFNs into a sparse MoE, then progressively distill from a larger MLLM teacher: first mimic (dense-to-dense then dense-to-sparse KL plus CE), then preference distillation that uses the teacher as the reference model in a KTO-style loss so the student can surpass the teacher on hallucination benchmarks.
Key results
- LLaVA-MoD-2B (Qwen-2-1.5B) averages 61.6 on comprehension benchmarks, surpassing Qwen-VL-Chat-7B by 8.8% using 5M samples and 23% trainable parameters.
- LLaVA-MoD-2B reaches 11.2% response-level and 5.9% mention-level hallucination rate on Object HalBench, beating RLHF-V (12.2% / 7.5%).
- Sparse MoE architecture beats dense student by 3.7% average gain under distillation; KD beats SFT by 8.1% average gain for MoE training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add preference distillation to mimic distillation (ablation) | LLaVA-MoD-2B · Object HalBench | response-level hallucination rate: 39.1 -> 11.4 | Helped |
| add preference distillation to mimic distillation (ablation) | LLaVA-MoD-2B · Comprehension benchmarks | AVG: 60.2 -> 59.9 | Hurt |
| use knowledge distillation instead of supervised fine-tuning (ablation) | MoE E4T2 · Comprehension benchmarks | AVG: 55.7 -> 60.2 | Helped |
| add dense-to-dense distillation before dense-to-sparse distillation (ablation) | Mimic distillation · Comprehension benchmarks | AVG: 57.9 -> 60.2 | Helped |
| add dense-to-dense distillation before dense-to-sparse distillation (ablation) | Mimic distillation · training cost | GPU hours: 1536 -> 960 | Helped |
| distill response tokens only rather than response and instruction tokens (ablation) | Mimic distillation · Comprehension benchmarks | AVG: 58.4 -> 60.2 | Helped |
| use sparse rather than dense architecture (ablation) | LLaVA-MoD distillation · Comprehension benchmarks | AVG: 57.1 -> 59.2 | Helped |
| use 4 rather than 8 experts with Top-2 routing (ablation) | MoE block · Comprehension benchmarks | AVG: 63.6 -> 64.0 | No clear effect |
| activate Top-2 rather than Top-1 experts (ablation) | MoE with 4 experts · Comprehension benchmarks | AVG: 62.6 -> 64.0 | Helped |
| use Qwen-1.5-7B rather than Qwen-1.5-4B as teacher (ablation) | Qwen-1.5-1.8B student · Comprehension benchmarks | AVG: 58.9 -> 60.2 | Helped |
| use Qwen-1.5-7B rather than Qwen-1.5-4B as teacher (ablation) | Qwen-1.5-0.5B student · Comprehension benchmarks | AVG: 53.7 -> 53.8 | No clear effect |
| use KTO rather than DPO for preference optimization (ablation) | LLaVA-MoD preference distillation · Object HalBench | response-level hallucination rate: 20.0 -> 11.4 | Helped |
Surprising findings
- Preference distillation with the teacher as reference lets the student surpass the 7B teacher on hallucination benchmarks (e.g., Object HalBench response rate 11.4 vs teacher 50.1 for Qwen-1.5-7B).
- KTO outperformed DPO for preference distillation, with DPO reaching 20.0 response hallucination rate vs 11.4 for KTO.
Limitations
- Teacher and student must share the same LLM family to keep a consistent vocabulary, limiting heterogeneous distillation.
- Loading both teacher and student requires substantial memory during distillation.
- Preference distillation improves hallucination benchmarks but gives inconsistent comprehension gains.
Experimental setup
Datasets: LLaVA-1.5-Pretrain ALLaVA-Caption-4V ShareGPT4V-PT MIMIC-IT LVIS LRV SViT ShareGPT4V-100K TextCaps LLaVA-1.5-Instruct GQA VQAv2
Benchmarks: GQA VizWiz ScienceQA-IMG TextVQA MME MMBench MMBench-CN Object HalBench POPE MMHal-Bench
Metrics: accuracy F1 response-level hallucination rate mention-level hallucination rate MMHal score
Baselines: Qwen-VL-Chat MiniCPM-V-2 VILA-3B Mini-Gemini-2B Bunny-2B MoE-LLaVA-2B SPHINX-Tiny RLHF-V RLAIF-V POVID HA-DPO
Builds on: LLaVA sparse upcycling KTO DPO MiniLLM
Closest papers by meaning
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR 2024 · similarity 0.88)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.88)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.88)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.87)
- A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.