Research map / Vision-language and multimodal / Multimodal LLMs
ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST3, a training-free MLLM inference acceleration framework that progressively prunes visual tokens across layers (PVTP, with QK heredity for "lazy layers") and anneals visual KV cache during generation (VTA), roughly halving FLOPs and doubling decoding speed on LLaVA.
Problem
Visual tokens in MLLMs (e.g., LLaVA) incur large inference cost; existing training-free pruning methods use coarse, layer-oblivious pruning and ignore how visual attention changes across generation steps, while retraining-based token reduction is resource-intensive.
Approach
Prune inattentive visual tokens progressively at strided layers based on last-token attention, reuse attention patterns in "lazy" layers via QK heredity, and cosine-anneal the retained visual KV cache as generated length grows.
Key results
- On LLaVA-1.5-7B, ST3 cuts FLOPs from 9.38T to 4.37T (53.41% reduction) and latency from 70.80ms to 40.12ms with MME 1866.71 vs 1861.97 for the original.
- On LLaVA-NeXT-13B, ST3 reduces FLOPs by 61.81% and latency by 68.35% (691.02ms to 218.71ms) while keeping accuracy close to the baseline.
- On captioning with KV cache, ST3 uses 2.27G memory vs 4.94G for LLaVA-1.5-7B while keeping CIDEr at 110.66 on COCO2017 (vs 110.43 original).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ST3 visual-token trimming instead of unpruned decoding | LLaVA-1.5-7B · Inference | Latency: 70.80ms -> 40.12ms | Helped |
| Apply ST3 visual-token trimming instead of unpruned decoding | LLaVA-1.5-7B · MME | Total score: 1861.97 -> 1866.71 | Helped |
| Apply ST3 visual-token trimming instead of unpruned decoding | LLaVA-1.5-7B · ScienceQA image subset | Accuracy: 69.46 -> 68.96 | Hurt |
| Use cosine VTA decay with τ=50 instead of exponential decay with σ=20 (ablation) | LLaVA-1.5-7B · COCO2017 caption validation | CIDEr: 110.29 -> 110.66 | Helped |
| Use cosine VTA decay with τ=50 instead of linear decay with τ=50 (ablation) | LLaVA-1.5-7B · Flickr30k test | CIDEr: 74.33 -> 74.91 | Helped |
| Set cosine VTA parameter τ=10 instead of τ=50 (ablation) | LLaVA-1.5-7B · COCO2017 caption validation | CIDEr: 110.66 -> 103.05 | Hurt |
| Use pruning stride S=1 instead of S=7 (ablation) | LLaVA-1.5-7B · Inference | Latency: 40.12ms -> 43.88ms | Hurt |
| Use pruning stride S=1 instead of S=7 (ablation) | LLaVA-1.5-7B · MME | Total score: 1866.71 -> 1836.47 | Hurt |
| Reuse layer 7 attention for layers 8∼10 (ablation) | LLaVA-1.5-7B · MME | Total score: 1866.71 -> 1480.70 | Hurt |
| Reuse layer 28 attention for layers 29∼31 (ablation) | LLaVA-1.5-7B · MME | Total score: 1866.71 -> 1863.54 | No clear effect |
| Reuse layer 28 attention for layers 29∼31 (ablation) | LLaVA-1.5-7B · Inference | Latency: 40.12ms -> 38.88ms | Helped |
Surprising findings
- VTW collapses on long-sentence captioning (e.g., CIDEr 67.20 on COCO2017 for LLaVA-1.5-7B), even though it performs well on single-token datasets.
- Applying QK heredity to shallow layers can severely degrade performance; some single-layer reuse configurations drop MME/AI2D to near zero.
Limitations
- Uniform pruning step size in PVTP; non-uniform schedules not explored.
- Evaluated only on short-generation tasks and captioning; longer-text generation not tested.
- Only LLaVA 7B and 13B models evaluated.
Experimental setup
Datasets: MME AI2D ScienceQA MMMU MMBench POPE COCO2017 caption Flickr30k NoCaps
Benchmarks: MME AI2D ScienceQA MMMU MMBench POPE COCO2017 caption Flickr30k NoCaps
Metrics: accuracy FLOPs decoding latency CIDEr KV cache memory MME total score
Baselines: FastV VTW
Builds on: FastV VTW LLaVA
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.94)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.94)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.93)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.92)
- Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.