Research map / Vision-language and multimodal / CLIP and zero-shot
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoPE-CLIP, a mask-free structured pruning framework for CLIP that scores heads, FFN neuron groups and layers by the zero-shot/task retrieval drop when removed (MoPE metric), supporting both pre-training and fine-tuning compression.
Problem
Large vision-language pre-trained models such as CLIP are too big for resource-constrained platforms, and existing VLP compression methods either use uni-modal pruning metrics that transfer poorly to cross-modal tasks or rely on costly mask-search or multi-stage distillation.
Approach
A Module-wise Pruning Error metric measures each head, FFN neuron group, or Transformer layer by the recall drop on cross-modal retrieval when it is removed; the metric builds cost tables that guide one-shot width and depth pruning, followed by distillation of cross-modal similarity, uni-modal features and hidden states from the teacher CLIP.
Key results
- MoPE-CLIP-base (86M vision + 42M text) reaches 52.8% TR@1 and 37.3% IR@1 on zero-shot MSCOCO, above ALIP pre-trained on YFCC15M by 6.0 and 8.0 points.
- MoPE-CLIP-base averages 58.8% zero-shot top-1 over 11 classification datasets when pre-trained on YFCC15M, surpassing the best prior efficient pre-training method, ALIP (41.8%).
- On a 0.5x width-pruned fine-tuned ViT-L/14 vision encoder (86M, 18 layers), MoPE-CLIP reaches 69.7% TR@1 on MSCOCO versus 67.6% for DynaCLIP at the same size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoPE-based structured pruning for the vision encoder | Fine-tuned CLIP-ViT-L/14 · MSCOCO image-to-text retrieval | TR@1: 70.3 -> 72.1 | Helped |
| Use MoPE-based text-encoder pruning | Fine-tuned CLIP-ViT-L/14 · MSCOCO text-to-image retrieval | IR@1: 59.3 -> 59.6 | Helped |
| Prune and pre-train the compact model using MoPE-CLIP | CLIP-ViT-L/14 · MSCOCO zero-shot image-to-text retrieval | TR@1: 46.8 -> 52.8 | Helped |
| Prune and pre-train the compact model using MoPE-CLIP | CLIP-ViT-L/14 · MSCOCO zero-shot text-to-image retrieval | IR@1: 29.3 -> 37.3 | Helped |
| Use MoPE-based module selection | MoPE-CLIPbase · 11 zero-shot image classification datasets, YFCC15M | average top-1 accuracy: 40.2 -> 58.8 | Helped |
| Use MoPE rather than top-layer selection (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, prune 6 layers at 0.375 width | TR@1: 57.6 -> 69.7 | Helped |
| Use MoPE rather than bottom-layer selection (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, prune 6 layers at 0.375 width | TR@1: 63.9 -> 69.7 | Helped |
| Use MoPE rather than every-other-layer pruning (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, prune 6 layers at 0.375 width | TR@1: 66.6 -> 69.7 | Helped |
| Use MoPE rather than loss-gradient layer selection (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, prune 6 layers at 0.375 width | TR@1: 66.3 -> 69.7 | Helped |
| Prune width before depth (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, 86M model | TR@1: 64.0 -> 69.7 | Helped |
| Prune width before depth rather than pruning both simultaneously (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval, 86M model | TR@1: 61.4 -> 69.7 | Helped |
| Add cross-modal similarity distillation loss Lsim (ablation) | MoPE-CLIPV · MSCOCO image-to-text retrieval | TR@1: 68.8 -> 69.7 | Helped |
Surprising findings
- Every-other layer pruning, a standard BERT depth-pruning heuristic, performs worse than other layer-selection rules when pruning six layers on CLIP.
- Hidden-state distillation fails for CLIP-ViT-B/32 and SE-CLIP because of patch-count mismatch with ViT-L/14, so distillation gains are diminished for those architectures.
Limitations
- Pre-training results use a small CC3M-scale dataset, and scaling behaviour on larger data is only partly examined.
- The MoPE metric is tied to the chosen retrieval proxy (Recall Mean on MSCOCO validation), so it may not transfer to other task types.
- Distillation of hidden states requires matching patch counts between teacher and student, which limits architectures.
Experimental setup
Datasets: CC3M CC12M YFCC15M MSCOCO Flickr30K
Benchmarks: MSCOCO Flickr30K CIFAR10 CIFAR100 Caltech101 Flowers102 OxfordPets DTD StanfordCars FGVC-Aircraft SUN397 Food101
Metrics: TR@1 TR@5 TR@10 IR@1 IR@5 IR@10 top-1 accuracy latency (ms)
Baselines: CLIP-ViT-B/32 SE-CLIP MagnCLIP DynaCLIP UPop-CLIP TinyCLIP EfficientVLM SLIP DeCLIP UniCLIP MCD ALIP
Builds on: CLIP DynaBERT UPop TinyCLIP MagnitudePruning
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.89)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.89)
- ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning (AAAI 2026 · similarity 0.89)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.88)
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.