Research map / Architectures and efficiency / Attention and transformers
PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PACT, a training-free visual token reduction method that prunes unimportant tokens with an attention-free importance metric (EUTI) and merges redundant ones via a new clustering algorithm (DBDPC) at an early LLM layer.
Problem
Visual language models incur high inference cost from large numbers of visual tokens that are often irrelevant or redundant, and existing reduction methods either rely on attention scores (incompatible with FlashAttention) or address only one of pruning or redundancy.
Approach
Combine an attention-free importance score (global query dot key softmax scaled by hidden-state norm) for pruning with Distance Bounded Density Peak Clustering, which bounds intra-cluster distances by a threshold, to merge the remaining tokens; pruned tokens close to cluster centers are recovered and merged in.
Key results
- On LLaVA-OneVision-7B, PACT reaches a 71.3% visual token reduction with only a 1.4% performance drop, while previous SOTA methods show at best a 4.4% drop at equal reduction.
- At a 98.6% Approach-to-Reference Metric Ratio on LLaVA-OneVision-7B, PACT achieves a 225% throughput ratio and 31% GPU memory reduction (19.05 GB vs 27.4 GB).
- PACT outperforms FastV, VTW, ToME, PruMerge and HiRED across LLaVA-OneVision-7B, InternVL2-8B, Qwen2-VL-7B-Instruct and LLaVA-1.6-Mistral-7B at equal reduction ratios and equal throughput.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PACT for visual token pruning and clustering | LLaVA-OneVision-7B · LLaVA-OneVision-7B benchmark suite | throughput ratio: 100% -> 225% | Helped |
| Use PACT instead of FastV at equal Approach-to-Reference Metric Ratio | LLaVA-OneVision-7B · LLaVA-OneVision-7B benchmark suite | visual token reduction ratio: 50% -> 71.3% | Helped |
| Combine pruning with clustering-based merging | PACT on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Helped |
| Replace DBDPC with other clustering algorithms | visual token clustering on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Use EUTI instead of attention-score-based FastV pruning | LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Helped |
| Use cluster centers as representatives instead of merging cluster tokens (ablation) | DBDPC on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Remove proportional attention (ablation) | DBDPC on LLaVA-OneVision-7B · higher reduction ratios | benchmark performance | Hurt |
| Assign each reduced token the mean position ID of its cluster (ablation) | DBDPC on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Assign sequential position IDs after reordering reduced tokens by mean position ID (ablation) | DBDPC on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Use hidden states instead of key vectors for clustering (ablation) | DBDPC on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Remove hidden-state norms from the EUTI importance score (ablation) | EUTI on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
| Remove the global-query score and use only hidden-state norms (ablation) | EUTI on LLaVA-OneVision-7B · multiple reduction ratios | benchmark performance | Hurt |
Surprising findings
- VTW shows a significant performance drop for reduction ratios above 40%, suggesting removing all visual tokens is only effective in later layers.
Limitations
- Relies on evaluation at a fixed early layer chosen by a key-distance heuristic rather than a learned policy.
- Clustering and pruning hyperparameters (cutoff distance, alpha, pruning percentage) must be tuned per model and reduction ratio.
Experimental setup
Benchmarks: AI2D TextVQA ChartQA DocVQA InfographicVQA MME MMBench MMVet MathVerse MathVista MMMU MMStar
Metrics: Approach-to-Reference Metric Ratio reduction ratio throughput ratio GPU maximum memory (GB) processing time
Baselines: FastV VTW ToME PruMerge HiRED agglomerative clustering k-means DPC DBSCAN
Builds on: Density Peaks Clustering ToMe FastV proportional attention
Closest papers by meaning
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.91)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
- Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.90)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.90)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.