Research map / Vision-language and multimodal / Multimodal LLMs
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies attention behaviors of MLLMs, finds three inference stages and that visual tokens stop contributing after text tokens absorb enough image information, then proposes DyVTE, a lightweight hyper-network that decides the layer at which all visual tokens are dropped, cutting FLOPs with little accuracy loss.
Problem
Visual tokens in MLLMs are highly redundant and inflate inference cost, and the optimal layer after which visual tokens can be removed varies across examples, tasks and models, so manual choice is impractical.
Approach
Use a small MLP hyper-network on text-token states to decide, per layer, whether to drop all visual tokens for the rest of inference; train it with binary labels from whether the early-exit answer matches the full-model answer, without backpropagating through the MLLM. Unlike layer-skipping early exit, text tokens keep running through all layers.
Key results
- DyVTE reduces LLaVA-1.5 7B computation by up to 45.7% on SEED without performance drop.
- Combined with FastV, DyVTE cuts LLaVA-1.5 computation by 51.5% with only 0.7% average performance drop.
- On SQA-I, DyVTE reduces LLaVA-1.5 13B FLOPs by 55.9% with -0.8% performance change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic visual-token exit | LLaVA-1.5 7B · SEED | TFLOPs: 9.2 -> 5.0 | Helped |
| Add dynamic visual-token exit | LLaVA-1.5 7B · SEED | accuracy: 58.6 -> 58.6 | No clear effect |
| Add dynamic visual-token exit | LLaVA-1.5 7B · Average (VL benchmarks) | TFLOPs: 9.2 -> 5.6 | Helped |
| Add dynamic visual-token exit | LLaVA-1.5 7B · Average (VL benchmarks) | accuracy: 67.0 -> 65.7 | Hurt |
| Add dynamic visual-token exit | LLaVA 7B · Average | accuracy: 55.7 -> 56.2 | Helped |
| Add dynamic visual-token exit | LLaVA 7B · Average | TFLOPs: 9.3 -> 5.5 | Helped |
| Combine dynamic visual-token exit with FastV | LLaVA 7B · Average | TFLOPs: 5.5 -> 4.5 | Helped |
| Use mean visual token status alone instead of mean text and last text (ablation) | DyVTE · Average | accuracy: 54.6 -> 55.0 | Mixed |
| Use last visual token status alone instead of mean text and last text (ablation) | DyVTE · Average | accuracy: 54.5 -> 54.0 | Mixed |
| Use mean text token status alone instead of mean text and last text (ablation) | DyVTE · Average | accuracy: 54.5 -> 54.6 | Helped |
| Use last text token status alone instead of mean text and last text (ablation) | DyVTE · Average | accuracy: 54.5 -> 54.0 | Hurt |
| Add last visual status to mean visual status (ablation) | DyVTE · Average | accuracy: 55.0 -> 54.9 | No clear effect |
Surprising findings
- Removing all visual tokens after a certain layer barely impedes performance, and in some cases (e.g., MM-Vet, TextVQA-like cases) DyVTE slightly improves accuracy over the default model.
- Using only attention scores as the exit state fails: the hyper-network exits too early and performance drops sharply.
Limitations
- Performs only full visual-token exit, not partial token retention in later layers, which may be suboptimal.
- Performance drops on some benchmarks (e.g., POPE for LLaVA-DyVTE 7B: 85.9 to 81.6; MMB for LLaVA 13B DyVTE drops ~1.7 points).
Experimental setup
Datasets: LLaVA-665k
Benchmarks: VQAv2 GQA ScienceQA TextVQA POPE MME MMBench SEED MM-Vet
Metrics: accuracy TFLOPs latency
Baselines: FastV ToMe LLaVA-1.5 7B LLaVA-1.5 13B
Builds on: FastV LLaVA-1.5
Closest papers by meaning
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.94)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.94)
- POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs (CVPR 2026 · similarity 0.93)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.92)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.92)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.