Research map / Vision-language and multimodal / Multimodal LLMs

ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViTCoP, a tuning-free three-stage visual token pruning framework that combines [CLS]-attention pruning in the vision encoder, VIC clustering plus K-norm elite selection and merging in shallow LLM layers, and K-norm pruning in deep LLM layers.

Problem

LVLMs incur high compute and memory cost from large numbers of visual tokens; existing pruning is either text-agnostic (losing task-relevant detail) or text-guided inside the LLM (yielding redundant selected tokens), and attention-based saliency is incompatible with FlashAttention.

Approach

Combine visual redundancy filtering in the vision encoder with step-wise visual-textual co-pruning in the LLM (density-peak clustering for diversity, K-vector L2 norm as a FlashAttention-compatible text-relevance proxy), using stage-specific strategies matched to shallow vs. deep LLM layers.

Key results

• At 192 retained tokens on LLaVA-1.5-7B, ViTCoP averages 98.5% performance retention versus 97.6% for VisionZip (the next best).
• At 64 retained tokens (88.9% pruning) on LLaVA-1.5-7B, ViTCoP keeps 94.7% average retention, surpassing VisionZip by 4.3 points and SparseVLM by 14 points.
• On LLaVA-NeXT-Video-7B at 88.9% pruning, ViTCoP averages 97.7% retention across four video benchmarks, versus 95.1% for SparseVLM.

Surprising findings

• Removing the deep-layer (stage III) pruning slightly improves COCO CIDEr, which the authors attribute to fine-grained detail loss.
• ViTCoP exceeds the unpruned model on some benchmarks (e.g., 100.3% on MMBench at 192 tokens, 100.8% on Video-MME).

Limitations

• Hyperparameters (d_c=8, tau=0.6) and pruning layers (2 and 22) were selected by grid search on COCO-2017 and may not transfer optimally.
• Stage II VIC clustering has O((pi1 m)^2) pairwise-distance cost, which grows with token count.
• Some baselines (FastV, PyramidDrop, SparseVLM) failed to run at 94.4% pruning on LLaVA-NeXT-7B, so the extreme-setting comparison is limited to VisionZip.

Experimental setup

Datasets: COCO-2017

Benchmarks: COCO-2017 Flickr30k GQA MMBench MME NoCaps OK-VQA POPE QBench ScienceQA VQA-v2 EgoSchema

Metrics: CIDEr accuracy exact match F1 retention rate TFLOPs GPU memory prefill latency

Baselines: FastV PyramidDrop SparseVLM VisionZip

Builds on: FastV PyramidDrop SparseVLM VisionZip FlashAttention

Closest papers by meaning

• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.93)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.93)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.93)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.92)