Research map / Vision-language and multimodal / Multimodal LLMs
Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills multiple visual experts (CLIP, ConvNeXt, EVA-02, optionally Pix2Struct/SAM) into a single CLIP vision encoder inside a LLaVA-1.5 VLM using teacher-specific and general-knowledge LoRA adapters with routers, plus coarse- and fine-grained distillation with token importance scoring.
Problem
Multi-expert vision encoders improve VLM visual understanding but incur large training and inference costs; naive multi-teacher distillation with a shared adapter set suffers from conflicting, noisy teacher knowledge.
Approach
Use teacher-specific LoRA adapters with a router so each adapter aligns with a single teacher, combined with general-knowledge LoRA adapters that learn a consensus summary of all teachers, and token-importance weighting of the distillation loss.
Key results
- HAWAII (7B) improves over LLaVA-1.5 on most benchmarks, e.g., MME 1540.2 vs 1510.7, SQA 70.5 vs 66.8, VizWiz 53.9 vs 50.0, MMBench 66.9 vs 64.3.
- HAWAII with SAM as extra teacher reaches 59.2 on TextVQA and 87.5 on POPE, the best in the main table.
- Ablation: teacher-specific FGKD raises average from 61.9 to 63.2; token scoring to 63.5; CGKD to 63.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill CLIP, ConvNeXt, and EVA-02 knowledge into a single vision encoder | LLaVA-1.5 with Vicuna-1.5-7B · VizWiz | score: 50.0 -> 53.9 | Helped |
| Distill CLIP, ConvNeXt, and EVA-02 knowledge into a single vision encoder | LLaVA-1.5 with Vicuna-1.5-7B · ScienceQA | score: 66.8 -> 70.5 | Helped |
| Distill CLIP, ConvNeXt, and EVA-02 knowledge into a single vision encoder | LLaVA-1.5 with Vicuna-1.5-7B · GQA | score: 62.0 -> 62.8 | Helped |
| Distill CLIP, ConvNeXt, and EVA-02 knowledge into a single vision encoder | LLaVA-1.5 with Vicuna-1.5-7B · MME | score: 1510.7 -> 1540.2 | Helped |
| Add token importance scoring to FGKD (ablation) | FGKD with teacher-specific LoRA adapters · Average | score: 63.2 -> 63.5 | Helped |
| Add token importance scoring to FGKD (ablation) | FGKD with teacher-specific LoRA adapters · GQA | score: 63.1 -> 62.8 | Hurt |
| Add CGKD and general-knowledge LoRA adapters (ablation) | FGKD with token scoring · Average | score: 63.5 -> 63.7 | Helped |
| Add Pix2Struct as a visual teacher (ablation) | HAWAII · VizWiz, GQA, MMBench | score: VizWiz: 53.9 -> 54.2; GQA: 62.8 -> 63.6; MMBench: 66.9 -> 67.1 | Helped |
| Add Pix2Struct as a visual teacher (ablation) | HAWAII · VQA Text, ScienceQA, SeedBench-I | score: VQA Text: 58.7 -> 58.6; SQA: 70.5 -> 69.5; SeedBench-I: 67.5 -> 67.2 | Hurt |
| Add SAM as a visual teacher (ablation) | HAWAII · VQA Text, VizWiz, GQA, ScienceQA, POPE, MMMU, SeedBench-I | score: VQA Text: 58.7 -> 59.2; VizWiz: 53.9 -> 54.3; GQA: 62.8 -> 63.3; SQA: 70.5 -> 70.8; POPE: 87.3 -> 87.5; MMMU: 36.6 -> 36 | Helped |
| Add SAM as a visual teacher (ablation) | HAWAII · MME | score: 1540.2 -> 1528.6 | Hurt |
| Increase general-knowledge adapters from 1 to 3 (ablation) | HAWAII · GQA | score: 62.6 -> 62.8 | Helped |
Surprising findings
- GQA slightly decreases with the teacher-specific and token-scoring components, which the authors attribute to GQA needing general rather than expert-specific knowledge.
- Adding more teachers (Pix2Struct, SAM) lowers some benchmarks such as MME and SQA/SeedBench, suggesting knowledge redundancy.
Limitations
- Only five pretrained vision experts were used due to compute limits.
- Evaluated only with Vicuna-v1.5-7B (and limited 13B and LLaVA-Next-7B) as the LLM.
- No error bars or statistical significance reported.
Experimental setup
Datasets: LLaVA-v1.5 pretraining data (558K) LLaVA-v1.5 instruction tuning data (665K)
Benchmarks: TextVQA VizWiz GQA ScienceQA POPE MME MMBench MMMU AI2D SeedBench
Metrics: accuracy MME perception+cognition score F1 (POPE)
Baselines: LLaVA-1.5 MoVE-KD BLIP-2 IDEFICS-9B Qwen-VL mPLUG-Owl2 InstructBLIP Video-LLaVA Eagle-X5 MoME MouSi Brave
Builds on: MoVE-KD LLaVA-1.5 LoRA DoRA LLaVA-Next
Closest papers by meaning
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.92)
- ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts (ICLR 2025 · similarity 0.90)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.89)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.