Research map / Architectures and efficiency / Pruning and efficient tuning

CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoreMatching, a training-free co-adaptive sparse inference framework that jointly prunes tokens (core tokens selected by neuron-activation overlap with core neurons) and FFN neurons (core neurons) in VLMs to speed up pre-filling and decoding.

Problem

VLM inference is costly in time and memory; token sparsity and neuron sparsity have been studied separately, and their interplay is unexplored, with token sparsity giving limited decoding speedups and neuron sparsity giving limited pre-filling speedups.

Approach

Core neurons (most frequently and strongly activated FFN neurons across all tokens of an input) are used to pick core tokens (tokens whose activated neurons overlap most with core neurons, cut by a maximum-geometric-distance knee threshold). A projection-guided criterion theoretically links token importance to activation overlap, so one pre-filling pass yields both token and neuron sparsity.

Key results

• On LLaVA-1.5-7B, CoreMatching keeps 64 image tokens (vs 576) with VQAv2 78.5, GQA 61.9, POPE 85.9 at 2.1 TFLOPs, matching or beating the original model on several tasks.
• On NVIDIA Titan Xp, CoreMatching reports 2.1x pre-filling speedup (1.05 s vs 2.23 s) and 9.2x decoding speedup (4.81 s vs 44.2 s) for LLaVA-1.5-7B with 610 input and 64 output tokens.
• On Qwen2.5-VL-7B, CoreMatching uses about 24 tokens on average and stays near-lossless (e.g., DocVQA 94.8 vs 95.7).

Measured effects reported in the paper

Surprising findings

• On MSRVT-QA and ActivityNet-QA, CoreMatching with Video-LLaVA exceeds the unpruned Video-LLaVA (59.8 vs 59.2; 48.4 vs 45.3).
• On AI2D with Qwen2.5-VL-3B-level experiments, CoreMatching surpasses the full-token model, suggesting pruning can remove noisy tokens.

Limitations

• Theoretical analysis relies on approximate orthogonality assumptions (W_q W_k^T ≈ θI, W_u/W_d approximately orthogonal) and a dominant self-attention term.
• Neuron-only latency baseline for PowerInfer is emulated by migrating a Llama2-7b predictor onto LLaVA, not a native VLM implementation.
• Some tasks show notable drops, e.g., TextVQA 54.5 vs 58.2 for LLaVA-1.5-7B, and LLaMA-3.2-11B results degrade without token pruning.

Experimental setup

Datasets: VQAv2 GQA SciQA TextVQA POPE MME MMBench SEED VizWiz MM-Vet DocVQA InfoVQA

Benchmarks: VQAv2 GQA SciQA TextVQA POPE MME MMBench SEED VizWiz MM-Vet DocVQA InfoVQA

Metrics: accuracy score TFLOPs latency (s) memory (GB)

Baselines: PruMerge+ FastV VoCo-LLaMA LLaVA-TRIM Dynamic-LLaVA PowerInfer PruMerge

Builds on: CoreInfer FastV PowerInfer PruMerge

Closest papers by meaning

• VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking (NeurIPS 2025 · similarity 0.91)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.91)
• The Sparsity Whisperer (NeurIPS 2026 · similarity 0.89)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.89)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.89)
• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.89)