Research map / Vision-language and multimodal / CLIP and zero-shot
MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task-Agnostic Vision-Language Pruning (TA-VLP) and MULTIFLOW, a gradient-free pruning method that scores weights by magnitude and information flow with multimodal layer-wise priors, prunes BLIP and XVLM once for multiple downstream tasks.
Problem
Existing VLM pruning methods are task-specific, requiring re-pruning from the dense model for every new downstream task; a single pruned VLM transferable to unknown tasks is needed.
Approach
Model each linear layer as a bipartite graph where weights are edges; score a parameter by the product of its magnitude and the saliency of its input and output neurons computed from calibration activations (gradient-free), then allocate layer-wise sparsity using modality-aware magnitude distributions from the pretrained VLM.
Key results
- At 63% sparsity on XVLM, MULTIFLOW improves text-retrieval R@1 over the second-best CHITA++ by +0.76% (up to +3.54% at 75% sparsity).
- At 90% sparsity, MULTIFLOW beats CHITA++ by +6.93 VQA accuracy, +10.14 BLEU@4 and +24.88 TR@1 on pruned VLMs while being about 41x faster on average.
- MULTIFLOW wins in 3 of 4 VQA instances and most ITR/IC settings across two VLMs and sparsities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MULTIFLOW instead of CHITA++ at 63% sparsity | XVLMCLIP · MSCOCO retrieval | Image-to-Text R@1: 76.59±0.20 -> 77.35±0.51 | Helped |
| Use MULTIFLOW instead of CHITA++ at 75% sparsity | XVLMCLIP · MSCOCO retrieval | Image-to-Text R@1: 70.33±0.04 -> 73.87±0.13 | Helped |
| Use MULTIFLOW instead of CHITA++ at 63% sparsity | XVLMCLIP · COCO Captions | BLEU@4: 37.82±0.20 -> 38.47±0.01 | Helped |
| Use MULTIFLOW instead of the strongest baseline at 75% sparsity | XVLMCLIP · VQA2.0 test-dev | accuracy: 74.00±0.01 -> 74.81±0.06 | Helped |
| Remove the imposed layer-wise distribution (ablation) | MULTIFLOW on XVLMCLIP · MSCOCO Image-to-Text retrieval, 63% sparsity | R@1: 77.35 -> 0.01±0.01 | Hurt |
| Remove multimodal priors from the imposed distribution (ablation) | MULTIFLOW on XVLMCLIP · MSCOCO Image-to-Text retrieval, 90% sparsity | R@1: 46.87 -> 35.55±0.26 | Hurt |
| Invert the pruning mask and keep the lowest-scoring parameters (ablation) | MULTIFLOW on XVLMCLIP · MSCOCO Image-to-Text retrieval, 63% sparsity | R@1: 77.35 -> 7.15 | Hurt |
Surprising findings
- BLIP collapses at high sparsity under all methods (near-random ITR at 75%+), while XVLM remains much more prunable, indicating different VLMs have different prunability.
- No single second-best pruning method exists across tasks and models; rankings change with task, model, and sparsity.
Limitations
- Large gap remains between pruned and dense performance, especially at high sparsity.
- Pruning results depend strongly on the VLM architecture; BLIP collapses under pruning at 90%.
- Evaluated only on two VLMs and three tasks, with unstructured sparsity.
Experimental setup
Datasets: CC3M Visual Genome MSCOCO COCO Captions VQA2.0
Benchmarks: MSCOCO image-text retrieval COCO Captions VQA2.0 test-dev
Metrics: R@1 R@5 BLEU@4 CIDEr VQA accuracy
Baselines: OMP LAMP SNIP ITERSNIP CHITA CHITA++ UPop random pruning
Builds on: one-shot magnitude pruning LAMP CHITA SNIP
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.91)
- Each Complexity Deserves a Pruning Policy (NeurIPS 2025 · similarity 0.91)
- ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning (AAAI 2026 · similarity 0.90)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.90)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.90)
- LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.