Research map / Vision-language and multimodal / Multimodal LLMs
Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EPIC, a progressive consistency distillation framework (token-wise TCD and layer-wise LCD) with shared-weight teacher-student passes that trains LLaVA-v1.5 to work with far fewer visual tokens without architecture changes.
Problem
Visual tokens make MLLMs expensive; training-aware token compression shifts the feature space and makes optimization hard, and prior methods often ignore this learning difficulty.
Approach
A single MLLM acts as both teacher and student via weight sharing; compression ratio (token-wise) and compression layer (layer-wise) are scheduled progressively from easy to hard, and a KL consistency loss from a slightly less-compressed teacher guides the student.
Key results
- With 128 visual tokens, LLaVA-v1.5 + TCD reaches 61.3 average vs. 61.4 for vanilla LLaVA-v1.5 (576 tokens), and LLaVA-v1.5 + LCD reaches 61.3 at 128 tokens.
- LLaVA-v1.5 + LCD with 256 tokens reaches 62.2 average (+0.8 over vanilla LLaVA-v1.5).
- At 64 tokens with FastV/DART/Random inference, FLOPs drop 83.9% and KV cache drops 88.9%; Random achieves about 1.6x CUDA-time speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Token Consistency Distillation with progressive compression | LLaVA-v1.5 with DART, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 61.4 -> 61.3 | No clear effect |
| Add Layer Consistency Distillation with progressive compression layer | LLaVA-v1.5 with DART, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 61.4 -> 61.3 | No clear effect |
| Add Token Consistency Distillation (ablation) | TCD, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 59.8 -> 61.3 | Helped |
| Use progressive compression ratios instead of a fixed 88.9% ratio (ablation) | TCD, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 59.1 -> 61.3 | Helped |
| Add Token Consistency Distillation (ablation) | TCD, retain 64 tokens · 10 visual understanding benchmarks | Avg. (%): 58.1 -> 59.4 | Helped |
| Use progressive compression ratios instead of a fixed 88.9% ratio (ablation) | TCD, retain 64 tokens · 10 visual understanding benchmarks | Avg. (%): 58.2 -> 59.4 | Helped |
| Add Layer Consistency Distillation (ablation) | LCD, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 60.5 -> 61.3 | Helped |
| Progressively shift compression from deeper to shallower layers (ablation) | LCD, retain 128 tokens · 10 visual understanding benchmarks | Avg. (%): 60.3 -> 61.3 | Helped |
| Add Layer Consistency Distillation (ablation) | LCD, retain 64 tokens · 10 visual understanding benchmarks | Avg. (%): 58.7 -> 59.4 | Helped |
| Progressively shift compression from deeper to shallower layers (ablation) | LCD, retain 64 tokens · 10 visual understanding benchmarks | Avg. (%): 57.8 -> 59.4 | Helped |
| Reduce visual tokens from 576 to 64 with EPIC + Random | LLaVA-v1.5-7B · POPE inference | CUDA Time (s): 1103.5 -> 697.3 | Helped |
| Retain 192 tokens with integrated token-wise and layer-wise progression | LLaVA-v1.5 + ICD · 10 visual understanding benchmarks | Avg. (%): 61.4 -> 61.7 | Helped |
Surprising findings
- Performance at 128-256 visual tokens matches or exceeds vanilla LLaVA-v1.5 with 576 tokens.
- Extreme compression (36 tokens) yields limited latency gains relative to 64 tokens, motivating a 'High ROI' regime analysis.
Limitations
- Applied only during the visual instruction tuning stage; pretraining-stage application is left to future work.
- Gains over vanilla LLaVA are small (about +0.0 to +0.8 average) at higher token budgets, and performance drops notably at 36-64 tokens.
- Code and data are not released at submission time.
Experimental setup
Datasets: LLaVA-665K
Benchmarks: VQA V2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MMBench-CN OCRBench
Metrics: accuracy average score (%) KV cache CUDA time FLOPs
Baselines: MQT-LLaVA QT-LLaVA LLaMA-VID VoCo-LLaMA TokenPacker LLaVA-Mini LLaVA-v1.5 Average-Pooling
Builds on: LLaVA-v1.5 DART FastV MQT-LLaVA
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.89)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.89)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.88)
- Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.88)
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.88)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.