Research map / Vision-language and multimodal / Multimodal LLMs

CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CATP, a training-free two-stage image token pruning method for multimodal in-context learning that combines pre-decoder alignment/diversity selection with progressive attention-difference and query-guided pruning inside shallow decoder layers.

Problem

Image token redundancy in interleaved multimodal in-context learning (many images per prompt) makes LVLM inference expensive, and existing single-image pruning methods (attention-based and diversity-based) cause large accuracy drops in this setting.

Approach

Prunes image tokens across the whole in-context sequence rather than per image: stage 1 scores tokens by image-text alignment plus facility-location diversity before the decoder; stage 2 uses attention differences across shallow layers and query-context similarity to progressively prune ICD tokens, then query image tokens at the next layer.

Key results

• At 77.8% image-token pruning on LLaVA-Next-7B 4-shot, CATP reaches 100.1% of vanilla average performance while FastV reaches 82.4% and DivPrune 86.3%.
• CATP averages 0.6% gain over vanilla across four LVLMs and eight benchmarks at 77.8% pruning, per the abstract.
• CATP reduces inference latency by 10.78% on average per the abstract; latency reductions on LLaVA-Next-7B, InternVL2.5-8B and Qwen2.5VL-7B range 8.0% to 16.5%.

Surprising findings

• Existing pruning methods (FastV, DivPrune) often underperform random pruning in 4-shot multimodal ICL.
• DivPrune increases latency (5.5% higher) on some models despite reducing FLOPs and KV cache, showing FLOPs are not a reliable efficiency indicator.

Limitations

• Hyperparameters (K, lambda1, lambda2) are tuned per model and the best of four configurations is reported for baselines, so comparisons depend on tuning.
• Gains over vanilla are small (about 0.6% average), and the efficiency gains are modest for some models (e.g., 8.0% on InternVL2.5-8B).

Experimental setup

Datasets: VQAv2 VizWiz OK-VQA GQA HatefulMemes MMStar MME-RealWorld VL-ICL

Benchmarks: VQAv2 VizWiz OK-VQA GQA HatefulMemes MMStar MME-RealWorld VL-ICL

Metrics: accuracy ROC-AUC average relative performance FLOPs inference latency KV cache memory

Baselines: FastV HiRED FitPrune VTW DivPrune SparseVLM PLPHP Random pruning

Builds on: FastV DivPrune FitPrune VTW SparseVLM

Closest papers by meaning

• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.91)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.91)
• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.90)