Research map / Vision-language and multimodal / Multimodal LLMs
MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoVE-KD, which distills multiple vision encoders (CLIP, EVA-02, ConvNeXt, optionally SAM) into a single student encoder inside LLaVA/LLaVA-NeXT using LoRA-based mixture-of-experts and CLIP [CLS]-attention-weighted distillation.
Problem
Using multiple visual encoders in a VLM boosts perception but multiplies computational cost; naive multi-teacher distillation into one encoder suffers from conflicts between teachers and forgetting.
Approach
Distill several teacher encoders into one student via encoder adapters, a mixture-of-LoRA-experts (MoLE) student encoder to avoid knowledge conflicts, and a distillation loss weighted at token level and teacher level by CLIP [CLS] attention.
Key results
- On LLaVA-1.5 7B, MoVE-KD-v1.0 raises average over eight benchmarks from 66.5 to 68.0 (ablation), with MMBench 64.3 to 66.3 and POPE 85.9 to 86.9.
- MoVE-KD-v1.1 (adding SAM-L teacher) reaches 79.9 VQAv2, 59.6 TextVQA and 67.4 MMBench on 7B, surpassing LLaVA-1.5 13B on several benchmarks.
- On LLaVA-NeXT 7B, MoVE-KD-v1.0 improves VQAv2 81.8 to 82.3, MME 1519.0 to 1537.2 and MMBench 67.4 to 67.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill multiple visual encoders with MoVE-KD-v1.0 | LLaVA-1.5 7B · VQA-V2 | accuracy: 78.5 -> 79.5 | Helped |
| Distill multiple visual encoders with MoVE-KD-v1.0 | LLaVA-1.5 7B · TextVQA | accuracy: 58.2 -> 58.3 | Helped |
| Add SAM-L as an additional teacher | MoVE-KD-v1.0 on LLaVA-1.5 7B · MME | score: 1524.5 -> 1509.1 | Hurt |
| Add SAM-L as an additional teacher | MoVE-KD-v1.0 on LLaVA-1.5 7B · TextVQA | accuracy: 58.3 -> 59.6 | Helped |
| Replace interpolation with encoder adapters (ablation) | LLaVA-1.5 7B with KD · Average (8 benchmarks) | average score: 66.3 -> 66.7 | Helped |
| Add MoLE (ablation) | LLaVA-1.5 7B with KD · Average (8 benchmarks) | average score: 66.7 -> 67.4 | Helped |
| Add token-level attention weights (ablation) | LLaVA-1.5 7B with KD and MoLE · Average (8 benchmarks) | average score: 67.4 -> 67.7 | Helped |
| Add adaptive teacher-level weights (ablation) | LLaVA-1.5 7B with token weights · Average (8 benchmarks) | average score: 67.7 -> 68.0 | Helped |
| Add MoLE without knowledge distillation (ablation) | LLaVA-1.5 7B fine-tuning · Average (7 benchmarks) | average score: 64.4 -> 64.5 | No clear effect |
| Unfreeze the vision encoder (ablation) | LLaVA-1.5 7B · VQA-V2 | accuracy: 78.5 -> 76.7 | Hurt |
| Set CLIP teacher weight to 0.8 rather than 0.6 (ablation) | MoVE-KD on LLaVA-1.5 7B · Average (8 benchmarks) | average score: 67.1 -> 68.0 | Helped |
| Vary CLIP teacher weight from 0.8 to 0.9 (ablation) | MoVE-KD on LLaVA-1.5 7B · Average (8 benchmarks) | average score: 68.0 -> 67.8 | Hurt |
Surprising findings
- Unfreezing the encoder without MoLE caused training to crash and degrade performance (e.g., VQAv2 78.5 to 76.7), so the gains are not from unfreezing.
- Distilling with SparseVLM-style text-based foreground weighting degraded performance, whereas CLIP [CLS] attention weighting helped.
Limitations
- Gains on VQA-Text are minimal or negative for some configurations, possibly because many questions are not vision-related.
- Teacher-weight allocation strategy is not fully explored; the authors say the optimal method needs further exploration.
- Pretraining-stage LLM scale shows diminishing returns from KD, suggesting the projector may be the bottleneck.
Experimental setup
Datasets: LLaVA Visual Instruct Pretrain LCS-558K LLaVA-1.5 fine-tuning data LLaVA-NeXT fine-tuning data
Benchmarks: VQAv2 GQA TextVQA VizWiz POPE ScienceQA MME MMBench
Metrics: accuracy MME perception score
Baselines: LLaVA-1.5 LLaVA-NeXT RADIO InstructBLIP Qwen-VL AM-RADIO
Builds on: LLaVA-1.5 LLaVA-NeXT AM-RADIO LoRA mixture of experts
Closest papers by meaning
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts (ICLR 2025 · similarity 0.90)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.89)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.89)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.88)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.