Research map / Vision-language and multimodal / Cross-modal learning
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MADTP, a vision-language transformer compression framework that aligns visual and language tokens via learnable tokens and uses alignment-guided, instance-adaptive token pruning to cut GFLOPs of CLIP and BLIP.
Problem
Vision-Language Transformers are computationally expensive due to many visual and language tokens, and existing token pruning methods are unimodal and static, ignoring cross-modality alignment and per-instance, per-layer adaptivity.
Approach
A Multi-modality Alignment Guidance module uses learnable tokens to align visual and language features and produce token attention maps, which together with class and self-attention scores guide a Dynamic Token Pruning module that learns a per-layer, per-instance threshold via sparsemax.
Key results
- On BLIP/NLVR2 at 0.8 reduce ratio, MADTP reaches 79.22% test accuracy with 26.46 GFLOPs (80% reduction) versus 83.08% uncompressed at 132.54 GFLOPs.
- At 0.5 reduce ratio on BLIP/NLVR2, MADTP achieves 82.85% test accuracy, 5.24 points above UPop.
- On CLIP COCO image-to-text retrieval at 0.75 ratio, MADTP R@1 is 66.2 vs 56.1 for UPop, at 92.4 GFLOPs (77% reduction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MADTP instead of UPop at reduce ratio 0.5 | BLIP · NLVR2 test | accuracy: 77.61 -> 82.85 | Helped |
| Add multimodality alignment guidance (MAG) (ablation) | MADTP on BLIP · NLVR2 test, reduce ratio 0.5 | accuracy: 80.96 -> 82.85 | Helped |
| Add dynamic token pruning (DTP) (ablation) | MADTP on BLIP · NLVR2 test, reduce ratio 0.5 | accuracy: 81.44 -> 82.85 | Helped |
| Use all three token importance scores instead of only Sself (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.49 -> 81.97 | Helped |
| Use all three token importance scores instead of only Stoken (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 80.68 -> 81.97 | Helped |
| Use all three token importance scores instead of only Scls (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.62 -> 81.97 | Helped |
| Set learnable-token count K to 50 instead of 100 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.44 | No clear effect |
| Set learnable-token count K to 150 instead of 100 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.49 | No clear effect |
| Set learnable-token count K to 200 instead of 100 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.74 | No clear effect |
| Set learnable-token count K to 256 instead of 100 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.79 | No clear effect |
| Set learnable-token count K to 512 instead of 100 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.79 | No clear effect |
| Set channel dimension dk to 50 instead of 768 (ablation) | MADTP on BLIP · NLVR2 dev, reduce ratio 0.5 | accuracy: 81.97 -> 81.44 | No clear effect |
Limitations
- Performance drops notably at high reduce ratios on some tasks (e.g., 0.8 ratio on NLVR2 loses about 3.9 points).
- Evaluation limited to CLIP and BLIP plus four datasets; generality to other VLTs is asserted rather than fully shown.
Experimental setup
Datasets: NLVR2 COCO Flickr30k VQA v2.0
Benchmarks: NLVR2 COCO Flickr30k VQA v2.0
Metrics: accuracy recall@K CIDEr SPICE GFLOPs
Baselines: UPop Static Token Pruning (STP) ELIP CrossGET DistillVLM
Builds on: UPop DynamicViT EVit ELIP CrossGET
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.91)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.91)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.91)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.91)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.