Research map / Vision-language and multimodal / Multimodal LLMs
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpecFlow, a training-free visual token pruning method for VLMs that diffuses CLS-attention energy over a kNN feature graph, allocates budgets with adaptive quadtree partitioning, and condenses pruned tokens into mean and residual sink tokens.
Problem
Visual token pruning in VLMs via Top-K attention scores under high compression produces spatially fragmented selections and loses contextual information, degrading accuracy.
Approach
Replace point-wise attention ranking with spectral heat-flow diffusion of CLS-attention energy on a kNN token-similarity graph, allocate per-region budgets via energy-guided quadtree partitioning, and append mean and residual sink tokens summarizing pruned tokens.
Key results
- LLaVA-1.5-7B with SpecFlow retains 95.6% of unpruned performance at 64 tokens (88.9% pruning), versus 94.9% for HoloV.
- At 192 tokens (66.7% pruning) SpecFlow reaches 98.7% relative average, the best among compared methods.
- On Qwen2.5-VL-7B at 64 tokens SpecFlow achieves 89.4% relative average vs 87.3% for HoloV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Diffuse energy on feature-similarity kNN graph instead of raw token self-attention (Token attn) (ablation) | SpecFlow token importance · GQA and TextVQA (64 tokens) | accuracy: +5.7 GQA, +4.6 TextVQA vs Token attn | Helped |
| Quadtree adaptive spatial partitioning instead of global Top-K selection (ablation) | SpecFlow token selection · GQA and TextVQA (64 tokens) | accuracy: quadtree reaches 55.3 GQA, 54.9 TextVQA; Top-K baseline values not given in text | Helped |
| Quadtree adaptive partitioning instead of uniform crops (Uni Crop) (ablation) | SpecFlow token selection · GQA and TextVQA (64 tokens) | accuracy: not quantified in text | Helped |
| Full SpecFlow pipeline vs HoloV at 192 tokens on LLaVA-1.5-7B | LLaVA-1.5-7B · 9 image benchmarks average | relative performance: 98.7% vs 98.2% | Helped |
| Full SpecFlow pipeline vs HoloV at 64 tokens on LLaVA-1.5-7B | LLaVA-1.5-7B · 9 image benchmarks average | relative performance: 95.6% vs 94.9% | Helped |
| SpecFlow pruning at 88.9% vs HoloV on Qwen2.5-VL-7B | Qwen2.5-VL-7B · MME, POPE, SQA, VQA-Text average | relative performance: 89.4% vs 87.3% | Helped |
| SpecFlow pruning vs FastV in prefill latency at 64 tokens (LLaVA-1.5-7B) | LLaVA-1.5-7B · end-to-end latency at 88.9% pruning | latency (s): SpecFlow 0.229s vs FastV 0.249s, but HoloV 0.221s is faster and SpecFlow prefill 15.5ms vs HoloV 14.0ms | Mixed |
| Sink tokens (mean + residual) appended to pruned set | SpecFlow · image benchmarks | accuracy: no ablation reported | No clear effect |
Surprising findings
- SpecFlow exceeds the unpruned LLaVA-1.5 baseline on MMBench and ScienceQA at 66.7% pruning.
Limitations
- Relies on access to vision-encoder CLS attention and intermediate representations, limiting use with black-box APIs.
- Quadtree assumes a 2D grid and does not model temporal structure for video.
- Compact mean and residual sinks may be insufficient for small objects, dense text, or counting.
Experimental setup
Datasets: GQA MMBench MMBench-CN MME POPE VizWiz ScienceQA VQA-v2 TextVQA MSVD-QA MSRVTT-QA
Benchmarks: GQA MMBench MMBench-CN MME POPE VizWiz ScienceQA VQA-v2 TextVQA MSVD-QA MSRVTT-QA
Metrics: accuracy relative average performance (%) score (video QA) latency (s) prefill time (ms) peak GPU memory (GB) FLOPs (T)
Baselines: ToMe FastV LLaVA-PruMerge PDrop HiRED VisionZip SparseVLM DART HoloV Video-LLaVA
Builds on: FastV HoloV VisionZip personalized PageRank SCLIP
Closest papers by meaning
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.89)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.89)
- SCoRe: Salience-Coverage Reduction for Vision Token Pruning in Vision-Language Models (CVPR 2026 · similarity 0.89)
- TwinPrune: Density-Aware Two-Phase Token Pruning for Vision-Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.