Research map / Vision-language and multimodal / Multimodal LLMs
A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGL, a training-free framework where a small VLM's all-layer aggregated attention guides visual token pruning in a large VLM, plus a small-VLM early-exit criterion that skips the large model on easy questions.
Problem
Large VLMs are costly at inference due to many visual tokens; single-layer attention used for token pruning is inaccurate at low retention ratios, while full-layer attention requires a costly full inference pass.
Approach
Use the globally aggregated attention map (summed over all layers and heads, prompt and generated tokens) from a small VLM of the same family as a proxy to prune visual tokens in the large VLM, and use the small VLM's confidence and prune-consistency score to decide early exit.
Key results
- On InternVL2-26B at 9% visual token retention, SGP reaches 89.58% score ratio versus 46.99% for FastV (TextVQA 78.98 vs 43.84).
- SGL pruning with 2B guidance keeps about 92-93% of 26B performance at 9% retention on TextVQA, SEED and RefCOCO, and up to 91% of visual tokens can be pruned.
- On Qwen2-VL-72B and LLaVA-OV-72B at 9% retention, guided by the 2B/0.5B models, performance stays at about 96% of the original (96.94% and 95.81%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small-VLM (InternVL2-2B) all-layer aggregated attention to guide pruning instead of single-layer FastV attention | InternVL2-26B visual token pruning · TextVQA, 9% token retention | score: 78.98 vs 43.84 | Helped |
| Use small-VLM guidance (SGP) instead of FastV at low retention | InternVL2-26B · RefCOCO, 9% token retention | score: 80.36 vs 19.65 | Helped |
| Attention from single-layer (FastV-style) pruning at low retention | InternVL2-26B visual token pruning · TextVQA, 9% token retention | score: 43.84 vs 82.45 original | Hurt |
| Aggregate attention from all layers of small 2B model rather than a single layer (ablation) | InternVL2-26B pruning guidance · TextVQA, 9% token retention | score: 78.98 vs 43.84 (one layer of 26B) | Helped |
| Increase fraction of small-model layers whose attention is aggregated (10% to 100%) (ablation) | small VLM attention guidance · TextVQA, 9% token retention | score: 44.40 at 10% layers to 78.98 at all layers | Helped |
| Include both prompt-token and generated-token attention in aggregation (ablation) | SGP attention aggregation · TextVQA / SEED / RC, 9% retention | score: TextVQA 78.98 vs 79.40 (last prompt token); RC 80.36 vs 83.51 (generated only) | Mixed |
| Use small VLM early exiting with combined confidence+consistency score (SEE) (ablation) | SGL early-exit decision · TextVQA early-exit score area | score: area 7.34 vs confidence 7.06 and entropy 7.04 | Helped |
| Use small VLM of different size/LM as guide (1B, 2B, 4B) (ablation) | InternVL2-26B pruning guidance · TextVQA, 9% retention | score: 79.38 (1B), 78.98 (2B), 79.70 (4B) | No clear effect |
| Scale large VLM to 40B and 76B with 2B guide | InternVL2-40B / 76B pruning at 9% retention · TextVQA | score: 92.38% and 92.98% of original score | No clear effect |
| Apply SGP with small Qwen2-VL-2B / LLaVa-OV-0.5B guides to large models | Qwen2-VL-72B and LLaVa-OV-72B · TextVQA, 9% retention | score: Qwen2-VL 82.88 vs 85.50; LLaVa-OV 75.98 vs 79.30 (about 96% retained) | Helped |
| Pruning with SGP at 64% and 35% retention on TextVQA (26B) | InternVL2-26B · TextVQA, 35% retention | score: 81.97 vs FastV 75.62 and ToMe 75.74 | Helped |
Surprising findings
- InternVL2-1B slightly outperforms InternVL2-2B as a guiding small model on three tasks.
- Using all-layer attention from the small 2B model slightly outperforms the 26B oracle on SEED.
Limitations
- Validated mainly on multimodal understanding tasks; not on unified understanding-and-generation VLMs.
- With the 26B model, SGP without SEE is slower than FastV and ToMe due to the small-VLM overhead.
- Small VLM must share architecture and tokenization with the large VLM to produce matching visual tokens.
Experimental setup
Datasets: TextVQA ChartQA DocVQA GQA SEED MMBench MM-Vet MME RefCOCO RefCOCO+ RefCOCOg
Benchmarks: TextVQA ChartQA DocVQA GQA SEED MMBench MM-Vet MME RefCOCO RefCOCO+ RefCOCOg
Metrics: accuracy score ratio relative to full model inference time peak memory FLOPs
Baselines: FastV ToMe InternVL2-26B (full) FastV-oracle
Builds on: FastV ToMe InternVL2
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.93)
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.93)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
- SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.92)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.92)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.