Research map / Vision-language and multimodal / Multimodal LLMs
EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EfficientLLaVA, an automatic structured-free pruning method for LLaVA-style LVLMs that searches layer-wise sparsity with 64 proxy PTB samples using an SRM-inspired generalization term and evolves the vision projector to enlarge the search space.
Problem
Automatic pruning of large vision-language models needs training data to pick per-layer pruning ratios, which is impractical given web-scale corpora; search on few proxy samples risks overfitting.
Approach
Searches layer-wise pruning ratios with evolutionary search on a fitness combining proxy-task loss and a Frobenius-norm-based Rademacher complexity (SRM) term, uses OBS for mask selection, and iteratively optimizes the vision projector weights, weighting candidates by neighborhood density, to evolve the search space.
Key results
- EfficientLLaVA reaches 83.05% on ScienceQA for LLaVA-SQA-7B at 0.5 pruning ratio vs 80.45% for SparseGPT.
- On LLaVA-v1.5-7B at 0.5 ratio, EfficientLLaVA gets 57.51% ScienceQA vs 55.25% SparseGPT, and 50.28% VizWiz vs 38.61%.
- Reports ~1.8x speedup on ScienceQA and 1.47x CPU speedup via DeepSparse with 0.45% accuracy drop on LLaVA-v1.5-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply generalizable pruning instead of uniform pruning | LLaVA-SQA-7B · ScienceQA | accuracy: 80.45 -> 83.05 | Helped |
| add generalization-aware policy search (SRM) | automatic pruning of LLaVA-SQA-7B · ScienceQA | accuracy: 81.34 -> 83.05 | Helped |
| add search space evolution (ablation) | LLaVA-SQA-7B pruning · ScienceQA | accuracy: 81.07 -> 83.05 | Helped |
| use candidate importance weights based on the upper bound rather than setting all c_m to 1 (ablation) | search space evolution for LLaVA-SQA-7B · ScienceQA | accuracy: 82.85 -> 83.05 | Helped |
| use EfficientLLaVA pruning instead of SparseGPT at pruning ratio 0.5 | LLaVA-v1.5-7B · ScienceQA | accuracy: 55.25 -> 57.51 | Helped |
| use EfficientLLaVA pruning instead of SparseGPT at pruning ratio 0.5 | LLaVA-v1.5-7B · Vizwiz | accuracy: 38.61 -> 50.28 | Helped |
| use EfficientLLaVA pruning instead of SparseGPT at pruning ratio 0.5 | LLaVA-v1.5-7B · MM-Vet | accuracy: 17.90 -> 31.70 | Helped |
| use EfficientLLaVA pruning instead of SparseGPT at pruning ratio 0.5 | LLaVA-v1.5-7B · LLaVA-Bench | accuracy: 48.80 -> 53.00 | Helped |
| increase the number of proxy samples beyond 64 (ablation) | pruning policy search · ScienceQA | accuracy | No clear effect |
| increase token length beyond 256 (ablation) | pruning policy search · ScienceQA | accuracy | No clear effect |
Surprising findings
- At 0.3 pruning ratio, the pruned LLaVA-v1.5-7B (66.07%) exceeds the dense model (64.70%) on ScienceQA, which the authors attribute to removing ambiguous neurons.
Limitations
- Speedup evaluated only on CPU via DeepSparse; no CUTLASS/GPU sparse acceleration tested due to equipment limits.
- Experiments limited to LLaVA-family models and VQA-style benchmarks.
- Method uses unstructured sparsity with OBS/SparseGPT masks, so practical speedup depends on kernel support.
Experimental setup
Datasets: PTB ScienceQA VizWiz MM-Vet LLaVA-Bench
Benchmarks: ScienceQA VizWiz MM-Vet LLaVA-Bench
Metrics: accuracy average pruning ratio speedup
Baselines: Uniform pruning Auto w/o Gen LLM-Pruner SparseGPT
Builds on: SparseGPT LLM-Pruner Optimal Brain Surgeon
Closest papers by meaning
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.92)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
- SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.91)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.91)
- ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models (CVPR 2026 · similarity 0.91)
- ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.