Research map / Vision-language and multimodal / Multimodal LLMs
ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECoFLaP, a two-stage coarse-to-fine pruning method for vision-language models that allocates per-layer sparsity using zeroth-order global importance scores and then prunes each layer with Wanda.
Problem
Layer-wise one-shot pruning of large vision-language models uses uniform per-layer sparsity and lacks global weight-importance information, while global pruning needs costly Hessian or gradient computation that is memory-prohibitive for LVLMs.
Approach
Estimate global importance scores via zeroth-order gradient approximation (forward-only, two perturbed forward passes with Gaussian noise), convert them into adaptive per-layer or per-block sparsity ratios, then apply Wanda layer-wise unstructured pruning at those ratios.
Key results
- At 0.5 sparsity on BLIP-2, zeroth-order ECoFLaP (w/ Wanda) reaches macro average 58.0 vs 57.2 for Wanda and 55.4 for gradient-based pruning, using about 8.93 GB vs 22.4 GB for the first-order variant.
- On LLaMA 7B at 0.6 sparsity, ECoFLaP with zeroth-order gradient lowers WikiText perplexity from 10.68 (Wanda) to 9.83, a 10.6% relative improvement.
- On BLIP NLVR2 at 0.5 sparsity without fine-tuning, ECoFLaP reaches 79.1 val vs 78.3 for Wanda and 76.9 for UPop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use globally informed adaptive layer sparsity with zeroth-order gradients instead of Wanda's fixed layer sparsity | BLIP-2 pruning · visual question answering, image captioning, and image retrieval | performance: 1.1%, 2.3%, and 1.0% | Helped |
| use zeroth-order rather than first-order gradients for global importance estimation | ECoFLaP · BLIP-2 pruning | GPU memory usage: 40% | Helped |
| use ECoFLaP instead of UPop | BLIP pruning at 0.5 sparsity · NLVR² and COCO captions | performance: 1.8% and 2.6% | Helped |
| use ECoFLaP with SparseGPT rather than SparseGPT alone | BLIP-2 · zero-shot multimodal tasks at 60% sparsity | macro average: 51.8 -> 53.1 | Helped |
| use 0.6 rather than 1.0 maximum per-layer sparsity (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 51.6 -> 58.4 | Helped |
| remove the maximum per-layer sparsity cap (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 51.6 | Hurt |
| increase calibration samples from 16 to 32 (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.2 -> 58.4 | Helped |
| increase calibration samples from 32 to 64 (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 58.3 | No clear effect |
| increase calibration samples from 32 to 128 (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 58.4 | No clear effect |
| increase noises per sample from 1 to 4 at fixed forward budget (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 58.1 | Hurt |
| increase noises per sample from 1 to 8 at fixed forward budget (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 58.0 | Hurt |
| increase noises per sample from 1 to 32 at fixed forward budget (ablation) | ECoFLaP · BLIP-2 VQAv2 val at 0.5 total sparsity | accuracy: 58.4 -> 58.1 | Hurt |
Surprising findings
- Using local layer-wise scores (from Wanda or SparseGPT) to set layer sparsities performs worse than uniform sparsity on CLIP.
- On GQA, the pruned BLIP-2 model answered 1072 questions correctly that the full model missed, while the full model answered 1219 correctly that the pruned model missed.
Limitations
- Max sparsity hyperparameter p_M must be set manually (p+0.1 chosen by ablation); the iterative alternative was avoided.
- Combination with SparseGPT gives smaller gains than with Wanda, and hyperparameter epsilon was barely tuned for LLaMA.
- Experiments are restricted to unstructured sparsity and mostly to BLIP-2 with zero-shot evaluation.
Experimental setup
Datasets: CC3M ImageNet C4 NLVR2 COCO captions VQAv2 OK-VQA GQA NoCaps Flickr30k MMLU WikiText
Benchmarks: VQAv2 OK-VQA GQA NoCaps Flickr30k NLVR2 COCO captions MMLU ImageNet-1k WikiText 11 CLIP zero-shot classification datasets
Metrics: accuracy CIDEr SPICE TR@1 IR@1 perplexity GPU memory usage
Baselines: Global Magnitude Pruning Gradient-based Pruning SparseGPT Wanda UPop
Builds on: Wanda SparseGPT WoodFisher UPop forward-forward algorithm
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.92)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.91)
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.91)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.90)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.