Research map / Vision-language and multimodal / Cross-modal learning
Exploring Intrinsic Dimension for Vision-Language Model Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies intrinsic dimension (TwoNN) across vision and language layers of VL pre-trained models and multiplies pruning importance scores by layer ID, yielding asymmetric vision/language pruning guidance.
Problem
Pruning vision-language models treats vision and language layers uniformly, although they differ in complexity and sensitivity; the paper asks whether modality-specific intrinsic dimension can guide layer-wise pruning.
Approach
Estimate per-layer intrinsic dimension with TwoNN on multimodal models and multiply existing weight importance scores (Magnitude, Gradient, PLATON) by the layer ID, so pruning favors removing weights from low-ID (language) layers and preserves high-ID (vision) layers.
Key results
- PLATON*ID at 80% pruning on MSCOCO captioning reaches CIDEr 129.2 vs 124.1 for PLATON alone (full model 133.3).
- At 40% pruning on NLVR2, Magnitude*ID raises accuracy from 52.9% to 79.8% (full model 83.4%).
- Language-only pruning at 40% drops CIDEr by only 1.9, whereas vision-only pruning drops it by 75.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Multiply PLATON pruning scores by layer ID (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 124.1 -> 129.2 | Helped |
| Multiply magnitude pruning scores by layer ID (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 84.9 -> 102.7 | Helped |
| Multiply gradient pruning scores by layer ID (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 47.2 -> 69.7 | Helped |
| Multiply PLATON pruning scores by layer ID (ablation) | NLVR2 model · NLVR2, 40% pruning | accuracy: 81.4 -> 81.5 | No clear effect |
| Multiply PLATON pruning scores by layer ID (ablation) | CLIP · Flickr30k, 40% pruning, text-to-image | Recall@1: 66.2 -> 71.7 | Helped |
| Multiply PLATON pruning scores by layer ID (ablation) | ViT-B/16 · CIFAR-100, 40% pruning | accuracy: 83.0 -> 82.8 | No clear effect |
| Multiply PLATON pruning scores by layer ID (ablation) | ViT-B/16 · ImageNet-1k, 80% pruning | accuracy: 70.5 -> 71.4 | Helped |
| Prune language-only rather than vision-only with PLATON*ID (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 110.5 -> 132.8 | Helped |
| Remove fine-tuning after magnitude pruning (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 77.6 -> 0.4 | Hurt |
| Remove fine-tuning after PLATON pruning (ablation) | BLIP · MSCOCO image captioning, 80% pruning | CIDEr: 124.1 -> 0.2 | Hurt |
Surprising findings
- Language-only pruning increases ID in later vision layers and cross-attention key/value layers.
- Magnitude pruning without fine-tuning yields the lowest ID values, and the highest layer ID may predict performance better than the final-layer ID in pruned models.
Limitations
- ID does not guarantee optimal dimensions; training or fine-tuning strategies for highly pruned models are not explored.
- ID improvements diminish as pruned performance approaches the full model, and ID is ineffective for very poorly performing models.
- Gains for Gradient and PLATON are mixed or marginal on some tasks, e.g., Gradient*ID on CIFAR-100 at 40%.
Experimental setup
Datasets: MSCOCO TextVQA Flickr30k NLVR2 CIFAR-100 ImageNet-1k
Benchmarks: MSCOCO NLVR2 Flickr30k CIFAR-100 ImageNet-1k
Metrics: CIDEr BLEU@4 METEOR ROUGE-L SPICE accuracy Recall@1 Recall@5 Recall@10
Baselines: Magnitude Gradient PLATON
Builds on: PLATON Movement pruning / cubic sparsity schedule TwoNN intrinsic dimension estimator Ansuini et al. 2019 ID profiling
Closest papers by meaning
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.89)
- Each Complexity Deserves a Pruning Policy (NeurIPS 2025 · similarity 0.89)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.89)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.88)
- IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.