Research map / Vision-language and multimodal / Multimodal LLMs

SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SwiftVLM, a training-free visual token pruning method that bypasses (forwards and re-evaluates) unselected visual tokens across dynamically chosen pruning layers, improving fine-grained VLM accuracy at fixed token budgets.

Problem

Visual token pruning in VLMs relies on early irreversible drop decisions, which discard tokens that become important in deeper layers and degrade fine-grained (localization) tasks.

Approach

A "bypass" pruning paradigm keeps unselected visual tokens (merged by similarity) and forwards them to the next pruning layer, where a hidden-state offset from the merged tokens realigns them for re-ranking; pruning layers are chosen via a dynamic programming procedure over a non-monotonic layer-wise selection-capability profile.

Key results

• At 128 retained visual tokens on LLaVA-1.5-7B, SwiftVLM reaches 96.7% of vanilla average accuracy versus 96.0% for FEATHER and 91.3% for FastV (relative accuracy).
• On RefCOCO at 192 tokens, SwiftVLM scores 66.6 versus 52.0 for FEATHER and 30.6 for FastV.
• On LLaVA-NeXT-7B with 22.2% tokens retained, SwiftVLM achieves 97.1% relative accuracy versus 87.5% for FEATHER.

Surprising findings

• Token merging hurts performance at the 192-token budget but helps at the 128-token budget.
• Token importance ranking across layers is non-monotonic in depth, with intermediate layers (around layer 15) peaking in selection capability.

Limitations

• Pruning layers are fixed per model via a profiling procedure on sampled data rather than adapted per input.
• Localization gains are largest; non-localization gains over baselines are small.
• Evaluated only on LLaVA-family models in the text.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg TextVQA GQA ScienceQA VQAv2

Benchmarks: RefCOCO RefCOCO+ RefCOCOg TextVQA GQA ScienceQA MME MMBench POPE

Metrics: accuracy relative accuracy FLOPs latency

Baselines: Vanilla FastV VisionZip PDrop SparseVLM FEATHER

Builds on: FastV PDrop FEATHER SparseVLM VisionZip

Closest papers by meaning

• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.93)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.93)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.93)
• IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.93)
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models (CVPR 2026 · similarity 0.93)