Research map / Vision-language and multimodal / Multimodal LLMs
Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UKMP, a structured pruning method for LVLMs (BLIP-2) combining a unified knowledge-maintenance importance metric (UKMI) with LoRA-based progressive distillation that recalls pruned weights (LPD).
Problem
Structured pruning of large vision-language models suffers from modality and block importance imbalance, unconverged frozen parameters that break gradient-based criteria, and loss of LLM knowledge, causing large performance drops at high pruning ratios.
Approach
Combines adaptive block-wise and modality-wise normalization, a Taylor importance using only the first term computed on poorly fitted next-token sub-tasks, and angular-distribution entropy injected into importance for the LLM; recovery uses a LoRA branch that recalls pruned weights plus a three-phase progressive distillation.
Key results
- At 50% pruning ratio, UKMP reaches 57.15 macro average vs 41.18 for the best baseline (UPop) and 39.67 for ECoFLaP sp.
- At 20% pruning ratio, UKMP achieves 63.40 macro average, slightly above the full BLIP-2 (3.9B) at 63.26 and above ECoFLaP sp at 62.54.
- At 50% pruning, UKMP's VQAv2 accuracy is 47.81 vs 18.53 for LLM-Pruner global; ablation shows UKMI lifts NoCaps CIDEr by 30.53 and LPD adds 3.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply UKMP instead of the strongest listed baseline | BLIP-2 ViT-g/14 + FlanT5XL · VQAv2 (50% pruning) | accuracy: 36.40 -> 47.81 | Helped |
| apply UKMP instead of the strongest listed baseline | BLIP-2 ViT-g/14 + FlanT5XL · OK-VQA (50% pruning) | accuracy: 10.50 -> 27.38 | Helped |
| apply UKMP instead of the strongest listed baseline | BLIP-2 ViT-g/14 + FlanT5XL · GQA (50% pruning) | accuracy: 25.15 -> 35.15 | Helped |
| apply UKMP instead of the strongest listed baseline | BLIP-2 ViT-g/14 + FlanT5XL · NoCaps (50% pruning) | CIDEr: 90.90 -> 96.92 | Helped |
| remove UKMI (ablation) | UKMP at 50% pruning · VQAv2 | accuracy: 18.53 -> 47.81 | Hurt |
| remove UKMI (ablation) | UKMP at 50% pruning · NoCaps | CIDEr: 62.86 -> 96.92 | Hurt |
| add UKMI (ablation) | UKMP at 50% pruning · NoCaps | CIDEr: 62.86 -> 93.39 | Helped |
| add LPD (ablation) | UKMP at 50% pruning · VQAv2 | accuracy: 39.01 -> 47.81 | Helped |
| add unifying on modules and modalities (UN) (ablation) | UKMP at 50% pruning with LPD · VQAv2 | accuracy: 29.22 -> 33.43 | Helped |
| add Taylor selection over next-token prediction (TS) (ablation) | UKMP at 50% pruning with UN · VQAv2 | accuracy: 33.43 -> 39.54 | Helped |
| add entropy injection (EI) for the LLM component (ablation) | UKMP at 50% pruning with UN and TS · VQAv2 | accuracy: 39.54 -> 47.81 | Helped |
| add weight recalling (WR) (ablation) | LPD with progressive learning at 50% pruning · VQAv2 | accuracy: 43.67 -> 47.81 | Helped |
Surprising findings
- At 20% pruning, the pruned BLIP-2 model outperforms the official full BLIP-2 3.2B model and even exceeds the original full model on OK-VQA and GQA, which the authors attribute to redundancy in LVLMs.
Limitations
- Evaluated only on one LVLM family (BLIP-2 with FlanT5XL) and zero-shot tasks.
- Entropy injection is applied only to the LLM component and reduces image captioning performance slightly.
- Relies on hard-threshold filtering of sub-tasks with a fixed threshold T and a small calibration set of 1000 pairs.
Experimental setup
Datasets: CC595K CC3M
Benchmarks: VQAv2 OK-VQA GQA NoCaps Flickr30k
Metrics: accuracy CIDEr SPICE TR@1 IR@1 macro average
Baselines: LLM-Pruner global LLM-Pruner local ECoFLaP sp FLAP UPop
Builds on: LLM-Pruner LoRA Taylor importance pruning ECoFLaP Torch-Pruning
Closest papers by meaning
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.91)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.90)
- ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models (ICLR 2024 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.