Research map / Vision-language and multimodal / Multimodal LLMs
Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Tokens Withdrawal (VTW), a training-free plug-and-play method that removes all vision tokens after a chosen deep layer of an MLLM, with the layer picked by a KL-divergence criterion on a tiny data subset, cutting roughly 40-50% of FLOPs while keeping accuracy.
Problem
MLLM inference is expensive because vision tokens dominate the input sequence length, and prior token pruning/merging methods (e.g., FastV, LLaVA-PruMerge) suffer from lack of flexibility across questions, unreliable importance metrics, and incompatibility with KV cache and FlashAttention.
Approach
Observes that attention sinks persist in MLLMs and that visual information migrates into text tokens within the first few layers, so vision tokens can be withdrawn wholesale at a single deep layer; the withdrawal layer is chosen by thresholding KL divergence of output logits on a tiny sample.
Key results
- On LLaVA-1.5-7B, VTW (K=16) cuts TFLOPs from 8.48 to 4.68 (55.19%) and scores MME 1872.43 vs 1866.10 baseline, while FastV reaches 1864.35 at 57.90% FLOPs.
- On LLaVA-NeXT-7B, VTW reduces latency per example from 0.52 s to 0.33 s and GPU per-sample memory by 35% on SQA-Image on an RTX 3090.
- On POPE, VTW (K=16) on LLaVA-1.5-7B scores 85.96 vs 85.83 baseline, whereas FastV drops to 82.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| withdraw vision tokens at layer K=16 (VTW) | LLaVA-1.5-7B · POPE | accuracy: +0.13 vs baseline (85.83 to 85.96); FastV 82.49 | Helped |
| withdraw vision tokens at layer K=16 (VTW) | LLaVA-1.5-7B · MME | score: +6.33 vs baseline (1866.10 to 1872.43) | Helped |
| withdraw vision tokens at layer K=16 (VTW) | LLaVA-1.5-7B · MMB_EN | accuracy: -0.09 (64.09 to 64.00) | No clear effect |
| withdraw vision tokens at layer K=16 (VTW) | LLaVA-1.5-7B · SQA_Img / AI2D / MMMU_Val | accuracy: SQA +0.05, AI2D +0.23, MMMU +0.70 vs baseline | Helped |
| FastV pruning after layer 2 instead of VTW | LLaVA-1.5-7B · POPE | accuracy: -3.34 vs baseline (85.83 to 82.49) | Hurt |
| random-layer withdrawal (K random in [8,24]) instead of selected K (ablation) | LLaVA-1.5-7B · MME | score: -124.76 vs VTW K=16 (1872.43 to 1741.34) | Hurt |
| random token discard of same count instead of vision token withdrawal (ablation) | LLaVA-1.5-7B · MME | score: collapse to 139.42 from 1866.10 | Hurt |
| withdraw vision tokens at layer 16 (VTW) | LLaVA-NeXT-7B · MME | score: +10.92 vs baseline (1846.33 to 1857.35) | Helped |
| withdraw vision tokens at layer 16 (VTW) | LLaVA-NeXT-7B · AI2D | accuracy: +0.04 vs baseline (65.31 to 65.35) | Helped |
| withdraw vision tokens at layer 16 (VTW) | Video-LLaVA · TGIF | accuracy: +0.02 vs baseline (0.25 to 0.27); FastV 0.21 | Helped |
| withdraw vision tokens at layer 16 (VTW) | Video-LLaVA · VideoMME overall | score: unchanged at 0.30 | No clear effect |
| keep original position embeddings of remaining text tokens (vs rearranging) after withdrawal (ablation) | LLaVA-1.5-7B VTW · SQA_Image / MMMU_Val / POPE | accuracy: SQA +0.10, MMMU +0.40, POPE +0.55 vs rearrange | Helped |
Surprising findings
- Withdrawing vision tokens at a random deep layer (VTW with random K) degrades MME severely (1741.34 vs 1872.43 for chosen K), showing layer choice matters.
- Randomly dropping the same number of input tokens (Rand) collapses performance to near zero on AI2D and MME, indicating that removing text tokens is harmful.
Limitations
- Gains on complex tasks are smaller because more layers may be needed for information migration.
- The withdrawal layer must be searched per model and task using a tiny subset of data.
- Evaluation on video and segmentation is limited in scope.
Experimental setup
Datasets: AI2D MMMU MME POPE
Benchmarks: AI2D ScienceQA-IMG MMMU-Val MMBench-EN POPE MME TGIF VideoMME ReasonSeg
Metrics: accuracy TFLOPs GPU memory latency gIoU cIoU
Baselines: FastV Random token dropping (Rand) LLaVA-PruMerge VTW with random layer
Builds on: FastV StreamingLLM LLaVA-1.5 KV Cache FlashAttention
Closest papers by meaning
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.94)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.94)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.93)
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.92)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.